FOR BI PROS LEARN Tableau
Reconfiguring Tableau Server without Restart: The Basics (Part1)
May 17, 2016
3
, , , , , ,

Reload Tableau ConfigurationTableau Software is continuously working on Server’s architecture to make it more robust, modern and enterprise grade. However, it still has its design heritage and technical dept that make some tasks too complex or uncomfortable. One of these is the way how minor and major configuration changes can be applied. Even if we made tiny changes like enabling repository’s readonly user or allowing peeps to access systeminfo.xml we do need a complete server restart.

Currently, the limitations of the out-of-the-box online reload are:

  • tabadmin service architecture (including tabspawn) and its current feature set
  • configuration template/file management
  • windows, as operating system (lack of posix signals)
  • unsophisticated traffic management between apache2 and tomcat services
  • reload hooks not implemented in webapps (like no support for JNI re-initalization)

Don’t worry if some of these points are not clear, we will discuss all of them in details sooner or later. In advance, there is a rumour that Tableau is working on Linux version of the Server. That will require a new tabadmin model as well as additional options to reload service configuration without restart using HUP signals in apache2 or redis for instance. Thus, when these changes happen my current post might be outdated I think at the moment it makes sense and helps to understand how Server operates.

Before we go back to theories let’s check a couple of examples and their relations with the above mentioned points.

Enabling External Access to the Tableau Server Database without Restart

This is a common task, practically one of the first steps after Tableau Server install. You cannot operate a Tableau Server without accessing its underlying database engines and cache layers (postgres, TDFS, solr, zookeper and redis) simply because not all of these information are exposed to user interfaces. What Tableau KB says about the process?

Enable Repository Access

Look at point three: tabadmin restart. Why would we need a restart to change a password? To answer this, lets first understand what dbpass command actually does.

First of all, make a copy of your <TableauDir>\data\tabsvc\config\pg_hba.conf and enable the readonly user with tabadmin dbpass --username readonly p@ssword  on one of the workers where you have repository.  Now compare the new pg_hba.conf with the old one:

Red lines are newly added after dbpass command

Red lines are newly added after dbpass command

As you see readonly is now allowed to logon from external locations. The unfortunate problem is that this file is not immediately reloaded. If you check the comments in pg_hba.conf it states:

This file is read on server startup and when the postmaster receives a SIGHUP signal. If you edit the file on a running system, you have to SIGHUP the postmaster for the changes to take effect. You can use “pg_ctl reload” to do that.

Hmm, pg_ctl. Can we just reload this config file with pg_ctl? Of course we can.

Now we can connect to the system, the new host based access file is active. Was restart required? No.

Although it was a very basic example it explains the usual problems around restarting:

  • After tabadmin commands we should understand which files are changed and which services are using those files (like pg_hba and postgres)
  • We should instruct the affected services to reload the configuration files or gracefully restart them

Now switch to application server layer to understand Tableau and Windows limitations, session routing and the tabspawn framework.

Enable Server Status URL from External IPs

The Monitoring Tableau Server KB page is pretty similar to the previous example. Change a parameter, then configure (dbpass calls configure internally as well) and finally restart.

In this case, as we know the configuration parameter’s name (wgserver.systeminfo.allow_referrer_ips) that refers to wgserver, but we all know that wgserver is past and systeminfo is managed by vizportal since a while (technical dept). Why is it not vizportal.systeminfo.allow_referrer_ips then? I don’t know, but I guess the guys have bigger issues to address so they’ve just left the parameter as it is.

Now back to theory and check what we have.

Tabsvc and Tabspawn

The typical layout for Tableau Server processes looks like:

Process tree

On the top you can find the tabsvc process. This one is executed from Windows services and it’s responsible for starting and stopping the system (I several times promised to server legend Paul Banoub to write about tabsvc and service monitoring, the details of this tabsvc belongs to that discussion). Now you can see that tabspawn basically executes its services inside a console host (for reasons like it can “press” ctrl-c for the apps). The fun part, if you kill a process it restarts immediately.

This is our vizportal process with its command line options. We can kill vizportal processes one by one - they will restarted immediately

This is our vizportal process with its command line options. We can kill vizportal processes one by one – they will restarted immediately

Let’s try it out quickly, let me add my father-in-law’s IP address to my test server with tabadmin, do a quick configure and kill all vizportal processes one by one.

It works as I expected, I was able to add my new IP without restarting the system. The only service kind that I restarted was vizportal. Victory? Yes and no.

Impact of force server restart

Going back to the previous example, the postgres server reload, the force process kill are something unacceptable there. All of us know that killing a process, especially the one that modifies persistent stores can cause issues. Luckily, with pg_ctl we were able to reload configuration without killing the processes. But with vizportal? What is the impact? Is there any other method than process kill?

Theoretically there is a way to reload applications inside tomcat container. If you install the manager application you might be able to reload or redeploy applications without restarting tomcat itself. Unfortunately Tableau’s applications are not supporting this.  You can try yourself, just deploy manager, try reload and check the service log files. It will be fully loaded with JNI related issues.

Going back to killing processes. Since in our particular case vizportal does not alter directly any state (it uses postgres and solr for instance) killing it should not cause any discrepancy on the persisent layer. That’s cool, at least we know that after killing the process vizportal will surely start.

Now, the impact from the user side. Any connections that were open (user clicked on something or an API call was in progress) could return error when the process kill occurs, otherwise any existing sessions will be redirected to an another vizportal session. Other sessions, were no requests were in progress (like the user just scrolled the portal, but not actually clicking on it) will survive the kill. How can we sure about this? Let’s have a look on the ProxyPass  rules for vizportal in apache2/gateway’s httpd.conf:

Two things to mention here.

  • First, for downloading workbooks and datasources might work as they handled by local gateway on the vizportal server. We do not have sticky sessions for web client, therefore the portal web UI requests can be fulfilled by any vizportal.
  • Second, we do have sticky sessions for vizportal api (used by desktop api, rest, xml, etc.), thus, in some circumstances it might cause issues. If __vizportal_upload_route  is set, then the request is forwarded to the same vizportal server for the consequent calls.

But what’s this __vizportal_upload_route ? After reading the redirect rules in httpd.conf it’s pretty obvious:

Ahum. What does this mean from our vizportal process kill perspective? In short: currently in-progress uploading sessions will be lost but other stuff will be fine.

If the gateway enforce sticky sessions then every session uses the same endpoint for all requests. The vizportal does not use other sticky session identified than the upload route, so the service is stateless except the uploads.

Graceful restart

OK, uploads and in-progress connections fail. Can’t we just do this restart in a graceful way without any unreasonable impact on the production? By default we cannot.

One option would be to wait until all uploads are done and all requests are served then stop the server. This is how SAP BusinessObjects or Cognos operates: their gateway stops sending new request to the “to be stopped” servers, then waits until the service complete all requests, then restarts.

Actually this is how we need to deal with more complex services like backgrounder or vizqlserver. Maybe in part2.

Summary

What a long post! We just covered the basics, did some magic with the repository (reload without restart) then look under the hood while reloading vizportal for new serverinfo or trusted authentication white list IPs.

If you have any comments or questions, you know, your box above this post is for you.

 

 

Tamás Földi

Tamás Földi

Director of IT Development at Starschema
Decades of experience with data processing and state of the art programming. From nuclear bomb explosion simulation to distributed file systems. ethical hacking, real time stream processing practically I always had a great fun with those geeky ones and zeros.
Tamás Földi
  • Toby

    How do you know which files are affected in the first place? Using your example, how does one know that pg_hba.conf is affected by dbpass?

    • realdataboss

      The best way it save the data/tabsvc/config folder before and after the changes and diff the files by contents.

  • Syed Rehman

    amazing info.. didn’t know you can get that deeper in tableauing!!!

Related items

/ You may check this items as well

Tableau Docker

HOWTO: Tableau Server Linux in Docker Container

Tableau Software is continuously working on Server...

Read more
Tableau Consistency Checker

Tableau Filestore Consistency Checker – How Repository Maps to Filestore

Tableau Software is continuously working on Server...

Read more
TCOT

Pushing the Limits of Tableau Server – Practical Lessons from a Zen Master

Tableau Software is continuously working on Server...

Read more