Making a working shiny server

Geeky stuff Shiny R packages R tricks R programming

This is very much work in progress and will probably always be that way.

Chris Evans https://www.psyctc.org/R_blog/ (PSYCTC.org)https://www.psyctc.org/psyctc/
2023-08-09

Show code 
### this is just the code that creates the "copy to clipboard" function in the code blocks
htmltools::tagList(
  xaringanExtra::use_clipboard(
    button_text = "<i class=\"fa fa-clone fa-2x\" style=\"color: #301e64\"></i>",
    success_text = "<i class=\"fa fa-check fa-2x\" style=\"color: #90BE6D\"></i>",
    error_text = "<i class=\"fa fa-times fa-2x\" style=\"color: #F94144\"></i>"
  ),
  rmarkdown::html_dependency_font_awesome()
)

Background

There is a huge amount of information about shiny on the internet. The place to start is definitely https://shiny.posit.co/. I’m not trying to replace any of the official information at posit.co (formerly Rstudio.com, much still in transition from that site to the new one), I’m just trying to document what I am learning as I struggle to get my head around shiny. This is something that I started trying several years ago and I did create a usable shiny app back then but realised that if I was going to provide something useful I needed to run my own shiny server. That I finally managed in early August 2023 and I am proving a slow learner though I suspect I am not alone in finding the shiny learning curve quite challenging. I hope this will help.

So what is shiny?

Well it’s actually “just” an R package with the crucial power that it enables you to embed R inside a web server so that you can provide interactive apps. See https://shiny.psyctc.org/ to see my expanding list of such apps. Clearly that means that shiny is a pretty impressive and no doubt huge lump of clever code! There are two versions: an open source one and a commercial one (or it may be various commercial versions, they’re all completely beyond my budget). However, the open source version is true open source software: anyone can download and use it without paying any licence fees (as long as you comply with the licence, which is easy for me).

Why am I doing this?

From time to time over the last few weeks I’ve asked myself that question as I’ve probe depths of my stupidity failing to get things working! My simple hope is that the apps I create will be useful, in principle to anyone but particularly for practitioners and researchers working in mental health (MH) and psychological therapies. That will probably mean that the apps will fall into three groups:

Creating a server

If you only want to create shiny apps to use on your own machine you don’t need a server, you can just write your apps and run them on your machine. The easiest way is probably to do it in Rstudio. However, I can’t see any reason anyone would do that: you’d just write the app as an ordinary R program!

So you want to offer your apps to others, what I want is for my apps to be usable for anyone on the internet. For that I needed a server sitting on a publicly visible IP address (the numeric address system that identifies machines on the internet) and it would also need a human readable address. So my server is shiny.psyctc.org currently on the IPv4 address 46.235.229.183 and the IPv6 address 2a00:1098:a6::1. This is actually a “virtual machine” running on a shared machine hosted by my excellent ISP, Mythic Beasts. I’ve been with them for 15 years now for my web servers (CORE, my non-CORE work and my personal site). They also run Jo-anne’s site and our Email. For a while in the last few years I had shiny running alongside my web servers on another VM but I worried that it is probably easy for a malevolent person to overload a shiny server and bring the machine hosting it to a halt so I gave that up in favour of the shiny server sitting on its own VM.

For now this server is Mythic Beasts’ “VPS 4” VMs: 2 CPU cores, 4Gb RAM and a 1Gb SSD drive. Looking at /proc/cpuinfo tells me that the cores are “Intel Xeon E312xx (Sandy Bridge)” running 2099.99 MHz (where does that number come from?!) with 16384Kb of onboard cache. It’s not super responsive but some of that may be because my broadband up in the Alps is pretty slow. I’ll watch the responsiveness as, I hope, it gets used more. [Update 28.iv.24: I only see a marginal improvement in responsiveness using the apps over fairly good broadband in London and the server is still used so little that user load is not an issue.]

Some configuring the server

Like me Mythic Beasts try to keep to open source software so the server is running Debian “bullseye” currently the “oldstable” release. Mythic Beasts saved me a lot of work setting it up with R and shiny but those were from the default Debian release version of R so I had to tweak the /etc/apt/sources.list.d to add a file, chris.list, saying:

deb http://cloud.r-project.org/bin/linux/debian bullseye-cran40/

to get R up to 4.3.1 (now). [Update 28.iv.24: now R 4.3.3 and I’ll probably shift it up to the brand new 4.4.0 shortly.]

To save myself updating the packages daily I created a little bash script /home/chris/updatePackages:

#!/bin/bash
nowDate=$(date +%F)
R CMD BATCH /home/chris/updatePackages.R $nowDate.Rout
mail -s "R update on shiny server $nowDate" shiny@psyctc.org < $nowDate.Rout

and this file, /home/chris/updatePackages.R

local({
  r <- getOption("repos")
  r["CRAN"] <- "https://cloud.r-project.org/"
  options(repos = r)
})

date()
update.packages(ask=FALSE,checkBuilt=TRUE)

set permissions on those files to 700 (i.e. only usable in any way at all to the owner, a bit of security). Then I added this line to crontab:

0 4 * * * /home/chris/updatePackages

so that cron (i.e. automatic) script gets R running, updates all the R packages that have new versions and Emails me the transcript. That crontab line gets that done every day at 04.00.

Backup of the apps

Mythic Beasts back up the entire VM daily so if, heaven forbid, the server dies or gets killed, they can always restore it to a state it was at at most 23 hours and 59 minutes (and some seconds!) earlier. My workflow creating my apps (next) ensures that there are always two mirror copies of all the apps actually before the third copy gets onto the server. This means that the only thing I’d ever lose would be the last day’s activity logs. I can’t see I’ll ever get so fascinated by the server usage that occasionally losing up to a day’s activity would bother me.

Logging the use of the server

Hm, with Mythic Beasts I have shiny using its default capability to keep a simple log of usage. There appear to be a number of other ways to log shiny usage but for now I’m just letting the default log file get built and I’ll write a bit of R to parse it soon. Details here when I do. [Update 28.iv.24: I have a simple logging system in use now, more below.]

My workflow for writing shiny apps (git)

One beauty of using shiny from within Rstudio is that I have all my shiny apps as a “project” in R and Rstudio makes it easy to link that project into a git repository. i have a love/hate relationship with git which is an open source “version control system”, i.e. it makes it easy to track changes across projects involving many files. This means that I can use git to do exactly that: to track the changes I make everytime I add a new app or tweak an existing one. Each time you make a change of any significance you “commit”, i.e. tell git to take that image. The beauty of git is that it keeps a record that you can unwind and it’s very economical of CPU time, storage and internet traffic making sure it only records changes, not taking copies of everything every time you change even a tiny bit.

Once you have your Rstudio project coupled to your local git image of the project it is then pretty easy to couple that git repository to github. Github is another wonder of the open source world: it is a huge repository for thousands of git (and, I think some other) packages mostly of software code. For the volumes of code I have it is free and very secure. I created my own github account (https://github.com/cpsyctc/) within which a copy of my git repository of my shiny server nests: https://github.com/cpsyctc/shiny-server/tree/master. This means that every time I commit changes and new apps locally using git from within Rstudio, I can then just tell git from within Rstudio to “push” the entire project/repository to github. I have set that up as a public repository on github which means that anyone who wants to can see and copy any of the code should they want. (I am using the MIT open source licence, details are in the github repository.) Then the final link is that I can “pull” that repository from github to the shiny server thus ensuring that by the time changes appear on the server I will already have the copy on my laptop and the copy on github. (And yes, the copy on the laptop will also get duplicated both locally and to the cloud within minutes of changes to that.) All nicely fast and with multiple copies in very different physical locations and accessible to anyone who wants any of this and free to them to get it provided that they have internet.

The pulling of the apps from github to the server is automated with another little bash script, /home/chris/cron_pull.sh:

#!/bin/bash
cd /srv/shiny-server
git pull

and the line:

*/15 * * * * /home/chris/cron_pull.sh

in crontab. That crontab line (aren’t they cryptic?!) runs the pull every fifteen minutes which is overkill really.

Love and hate with git and github

I did say I have a love/hate relationship with git. As long as you only let it do its thing it’s wonderful: I’m in love. However, if I ever forget the sequence and, say, be so stupid as to tweak anything inside the git repository but tweaking it directly on the shiny server then git recognises that things are out of synch and stops doing its thing. That’s actually 100% wise but the error messages and what you have to do to repair the mess you created are both not easily comprehensible for me and I slip from love to hate. I have, belatedly, realised that if I ever do that, the trick is to wipe the git repository on the server and just run the cron script to recreate the copy. However, if you get the github repository out of synch with the local one then things get messier and I find the github documentation can be pretty arcane. The simple rule is never change anything other than on your local, primary (“master”) copy.

My learning curve about programming shiny apps

I’ve put all that up there mostly just to remind myself what had to be done just to get a working shiny server and sensible workflow. Perhaps it will be useful to others. However, now we come to the crunch: writing shiny apps.

Structure of the shiny project

There are at least two ways of organising apps, both involve having a directory “apps” off the root of the shiny server. The apps each have their own directory inside the apps directory and, for now at least, I am using the way of creating the apps that puts all the code into one file always called app.R (which can make it easy to lose which you have open so I put the name of the app, which is the name of the directory, as a comment at the top of all my app.R files).

How I now think about each app

[I suspect this will evolve a lot. This is as of 26.viii.23, now 28.iv.24!]

The crucial thing to understand that seemed to take me a while to really accept is that though you are using R and lots of the code is exactly as it would be in an R or even Rmarkdown file it’s probably best to keep telling yourself that because you are constructing something quite different from the linearity of an R or Rmd file: thinks simply don’t read from top to bottom as they (mostly) do in R/Rmd files. I had to let go of some of my habits. The key thing for me has been to learn to “think reactivity”.

Reactivity

Key parts of shiny app code that enable reactivity

At the moment it is helping me to think of my apps having three parts that aren’t in my usual R code.

I am starting to think of an app as having those shiny specific bits and these “not-shiny” parts:

The key thing that I’m still not adjusting to well is that I can’t make my usual assumption that I can read from the top of the file downwards to understand the order in which things happen. I am also finding it hard to adjust to the fact that the structure of any app.R is like this:

### this is my convention of putting the name here
### CIcorrelation
### 
### load packages
library(shiny)
library(shinyWidgets)

# Define UI for application that does the work
ui <- fluidPage(
  setBackgroundColor("#ffff99"),
  ### this is from
  ### https://stackoverflow.com/questions/51298177/how-to-centre-the-titlepanel-in-shiny
  ### and centers the first title across the whole page by tweaking the css for head blocks
  tags$head(
    tags$style(
      ".title {margin: auto; align: center}"
    )
  ),
  ### to me it's a bit ugly that I end up putting the title here
  tags$div(class="title", titlePanel("Confidence interval for a Pearson or Spearman correlation\n\n")),
  
  # Get input values
  sidebarLayout(
    sidebarPanel(
      p("This shiny app is one of a growing number in ..."),
      ### more of the text cut from here for this Rblog post
      ###
      ### now we get set up the interface bit of the inputs
    numericInput("n",
                 "Total n, (zero or positive integer)",
                 value = 100,
                 min = 0,
                 max = 10^9,
                 width="100%"),
   ### others cut to make Rblog post shorter
  ),
  
  ###
  ### again, I am struggling to remember that the outputs, really the output placeholders go here
  mainPanel(
    h3("Your input and results",align="center"),
    verbatimTextOutput("res"),
    ### more snipped and you often have a number of outputs, text, tables, plots ...
  )
)
)


# Define server logic required
### this is the standard shiny server constructor
### the input and output arguments are vital, the session argument is optional but I think always wise
server <- function(input, output, session) {
  ### 
  ### start with validation functions
  ### I dropped the ones I had because I could use numericInput() to set the ranges
  ### but you might need functions to check relationships between inputs (say)
  
  ### 
  ### now the functions adapted from CECPfuns plotCIcorrelation
  ### I think I would do this differently now
  getCI <- function(R, n, ci = 0.95, dp = 2) {
    z <- atanh(R)
    norm <- qnorm((1 - ci)/2)
    den <- sqrt(n - 3)
    zl <- z + norm/den
    zu <- z - norm/den
    rl <- tanh(zl)
    ru <- tanh(zu)
    ci.perc <- round(100 * ci)
    retText <- paste0("Given:\n",
                      "   R = ", R,"\n",
                      "   n = ", n,"\n",
                      "   observed correlation = ", round(R, dp),
                      "\n",
                      "   ", ci.perc, "% confidence interval from ", round(rl, dp),
                      " to ", round(ru, dp),"\n\n")
    return(retText)
  }
  
  output$res <- renderText({
    validate(
      ### I have just left this in to demonstrate how validate works which seems OK
      ### need(checkForPosInt(input$n, minInt = 0), 
      ###     "n must be a positive integer > 10 and < 10^9"),
    )
    ###
    ### this is just passing the input variables to the function above
    ### I could just as well have put the arguments directly into that function
    getCI(input$R,
          input$n,
          input$ci,
          input$dp)
  })
}

### this bit is at the end of all shiny apps and puts together the two objects constructed earlier
# Run the application (ends all shiny apps in the one file, app.R format)
shinyApp(ui = ui, server = server)

So the structure is always this.

### name
### setup stuff
### define the user interface (doesn't have to be fluidPage but mine are so far)
ui <- fluidPage(
   ### general text and aesthetics
   ### input stuff
   ### output framework/placeholders
)

server <- function(input, output, session) {
   ### any validate functions to do validation not done by the input settings
   ### functions (I think they could go outside the server definition, not sure)
   ###    the ones that use inputs (e.g. input$n) actually build something active into the server
   ###
   ### define any reactive objects (none in example above, it's too simple to need them)
   ###
   ### named outputs like, here the "res" links to the quoted "res" in the ui
   output$res <- renderText(
      ### validation if there is some goes here so no output construction is attempted 
      ### if the input isn't OK.  Seems a late place to put it but that's because I'm 
      ### still thinking in a linear, top to bottom way
      validate(need(test),
         "error message for failed validation")
   )
}

### do the business
shinyApp(ui = ui, server = server)

I guess I am getting to understand this and I think it’s helped me to do that sort of anatomising of a very simple app.R down sort of in two stages: first a simplified state but as it were with the muscles and main organs exposed, then stripped to just the skeleton.

[Update: 28.iv.24] That is broadly still true, it has to be, however, I am now, slowly, adjusting to “shiny layout thinking” and I have also put quite a bit more complexity into my more recent apps, including logging of usage. I’ve also learned quite a bit more about different ways of formatting output so some of my app code would look quite a bit different from that now but I won’t change that as the basic ideas about the layout are still true.]

Inputs

As I am learning, these are defined in the construction of the ui object and there are nice functions for simple inputs with very useful direct validation such as min and max for numeric input. They all have the names input$name1, input$name2 (except that you want more sensible names than “name1” and “name2”, here they are “input$n”, “input$alpha” etc.) It helps me to think of “input” essentially as a named list and it’s a reactive list in the sense that if the user changes any input, the content of that slot in “input” changes instantly.

Reactive data/objects

You may or may not have reactive objects. These are like any object in R but crucially they change value the moment that any of their components are changed in the input. To that extent it’s sensible to think of them more as functions than as objects and they are constructed like functions:

reactiveSausage <- reactive({
   ### a sausage is made by stuffing stuffing into skin
   sausage <- makeSausage(input$stuffing, input$skin)
   sausage
})

OK, I know it’s a silly example and this assumes that makeSausage() is an ordinary function that has been defined or supplied from a loaded package somewhere in app.R. Technically I don’t have to construct the sausage object there and then return it by having it as that last line I could just have had

reactiveSausage <- reactive({
   ### a sausage is made by stuffing stuffing into skin
   makeSausage(input$stuffing, input$skin)
})

Reactive objects are used like functions with no arguments so if I want to print that reactiveSausage it’s:

print(reactiveSausage())

But of course, we don’t just print() things in shiny apps. (Well, sometimes we do just for debugging to get that on the console of the machine running the app.) That brings us to outputs.

Outputs

The key thing about outputs of a shiny app is that they are always the consummation of two things: the placeholder of the correct type put in the ui by something like

verbatimTextOutput("res"),

and the output constructor in the server, something like this:

output$res <- renderText(),

The “$res” in output maps to the quoted “res” in verbatimTextOutput("res") in the ui. I am starting to find it helpful to think of output\$, rather as I now think of input\$ as a sort of reactive list whose named members are accessed and used in the ui. I now think that creating outputs that work needs three things really:

Logging server use

Using the shiny default logging

Shiny comes with the option to log accesses. The configuration is stored in the very simple config file /etc/shiny-server/shiny-server.conf (on Linux):

# Instruct Shiny Server to run applications as the user "shiny"
run_as shiny;
access_log /var/log/shiny-server/access.log combined;

# Define a server that listens on port 3838
server {
  listen 3838;

  # Define a location at the base URL
  location / {

    # Host the directory of Shiny Apps stored in this directory
    site_dir /srv/shiny-server;

    # Log all Shiny output to files in this directory
    log_dir /var/log/shiny-server;

    # When a user visits the base URL rather than a particular application,
    # an index of the applications available in this directory will be shown.
    directory_index on;
  }
}

I have pushed my log format to “combined” from “default” to try to get more information and I’ve written an R script to parse the log and that should get a bit more interesting as things go forward in which case I’ll expand this bit here or create a new post for it.

Logging by building that into the apps

I’m glad I’ve started parsing the simple access.log. However, it’s clear that to get more information about use you have to build things into your apps to get that and there appear to be a number of different options, probably at least some of them incompatible with each other and of course each will slow the app and server down a little and some look to be pretty thin on documentation usable for someone like me and at least some look not to have been updated for some time. (Of course, in the open source world, sometimes that’s absolutely fine: the code is simple and sound, pretty much independent of other things (e.g. any changes to shiny) and hasn’t needed any changing for ages … but too often it’s a clue that the developer doesn’t have time to look after it or that it has been orphaned.)

[Update 28.iv.24] However, I am now logging use of all my apps using shiny.telemetry.

Here’s the code I’m using

suppressMessages(library(shiny.telemetry)) # load the package (the suppressMessages() is not strictly necesssary, some people seem to like it)

### 1. Initialize telemetry with default options (store to a local logfile)
### goes before ui section of the app
### I have stored the sqlite database in the shiny server root hence the "../../" that  points back from the app directory to the database
### make sure you remember to change the app_name if copying and pasting from another script!  This is the crucial bit that identifies the app
telemetry <- Telemetry$new(app_name = "Histogram_and_summary1",
                           data_storage = DataStorageSQLite$new(db_path = file.path("../../telemetry.sqlite")))

### lines skipped

### this next bit comes immediately in the uk
ui <- fluidPage(
  use_telemetry(), # 2. Adds the necessary Javascript to Shiny

### lines skipped

server <- function(input, output, session) {

### this actually starts the session logging
  telemetry$start_session(track_inputs = TRUE, track_values = FALSE) # 3. Track basics and inputs and not the input values

Loading the package and then those three lines are all that’s needed. I should have made notes at the time but my recollection is that setting up the sqlite database was easy enough once I had worked out to locate it in the root of the shiny server. The crucial thing is to put it in your .gitignore file in the root of the project/git repository, like this:

.Rproj.user
.Rhistory
.RData
.Ruserdata
telemetry.sqlite
tmp*
*in_progress*

Files named in .gitignore are, as the file name suggests, ignored by git so this ensures that I can have that database on my local machine so I don’t get errors complaining that the database doesn’t exist while developing apps locally. However, having it named in .gitignore ensures that I don’t overwrite the definitive database on the server every time I push changes up to it.

Sadly, but I think wisely, my shiny server is behind a proxy system on the server that ensures that everything is accessed via https so increasing the security of the server. However, this has the side effect that the IP addresses of users aren’t passed through to shiny so I can’t track individuals’ use and I only get aggregated data. I didn’t find the shiny app that comes in the shiny.telemetry app much use partly because it is designed for the situation in which you can distinguish between users, I wrote my own little Rmarkdown file that gives me some things that I find informative. That runs once a day and can be found here. I do track values for some apps and do analyse those for one app: RCI1 but I can’t see me getting much out of that sort of use tracking.

Visit count

page counter

Last updated

Show code 
cat(paste(format(Sys.time(), "%d/%m/%Y"), "at", format(Sys.time(), "%H:%M")))
19/04/2024 at 11:11

Reuse

Text and figures are licensed under Creative Commons Attribution CC BY-SA 4.0. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".

Citation

For attribution, please cite this work as

Evans (2023, Aug. 9). Chris (Evans) R SAFAQ: Making a working shiny server. Retrieved from https://www.psyctc.org/R_blog/posts/2023-08-25-making-a-working-shiny-server/

BibTeX citation

@misc{evans2023making,
  author = {Evans, Chris},
  title = {Chris (Evans) R SAFAQ: Making a working shiny server},
  url = {https://www.psyctc.org/R_blog/posts/2023-08-25-making-a-working-shiny-server/},
  year = {2023}
}