-Things to consider when writing statistical commentary
-
-
-
-
Introduction
-
-
DfE publishes over 60 distinct Official Statistics collections each year, as well as numerous experimental and ad-hoc releases.
-
The layout, structure, and length of these releases vary significantly across the Department - making it confusing and hard to navigate across releases.
-
When writing content to a general audience, statisticians must explain the answers in words that the audience can understand.
-
Migration to a new statistics dissemination platform - Explore Education Statistics (EES) - presents an opportunity for publications to be reviewed and for the way this is communicated to be more consistent and accessible.
-
Teams should continually improve their publications, assessing against best practice using our content checklist.
-
All of our publications should be designed around our users first and foremost, and you should consider this page in conjuction with any user engagement activities you are carrying out.
-
We are running publication scrums where teams can put forward their publication(s) for review by a group of volunteer analysts, providing feedback as ‘unfamiliar new readers’ to discuss ideas for further improvements. For more information, see the publication scrums section.
-
-
-
-
Checklist
-
-
The guidance on content design is in the form of a handy checklist, co-produced with local statisticians, and supported by the Good Practice Team. This builds on ONS’s Best Practice guidance on Data Visualisation and Writing about Statistics.
-
The content checklist is for teams to use throughout their project cycle, so that good content design is at the heart of what they deliver at all stages, rather than considered late in the process - Content design checklist (.xlsx).
-
There is also a dashboards version of the content design checklist, that runs through a number of things to think about when developing dashboards to compliment official statistics - Dashboards checklist (.xlsx).
-
-
-
-
Publication summaries
-
-
Publication summaries are a key tool in helping users find the statistics that they’re looking for. We use them on gov.uk pages and in the EES find statistics page.
-
Like on gov.uk, you only have 181 characters – to make sure you are fully utilising these, have a look through the following advice:
-
-
Use plain language to use terms and phrases that users are likely to use e.g. gender pay gap versus Annual Survey of Hours and Earnings.
-
Is it clear what the geographical coverage of this publication is e.g. England?
-
Is it clear how frequently the releases are published?
-
Is it clear what breakdowns you cover? E.g. Ethnicity, Gender, SEN?
-
Include the abbreviations but make sure to also write them out in full so that people can search for either, e.g. Free School Meals (FSM)
-
Avoid phrases like “This release covers” as this wastes characters and delays users getting to the main information.
-
Have you looked at the EES analytics to see what users key search terms on your publication are? Are key words front loaded In your summary?
-
Don’t waste space on including definitions of a topic within the summary.
-
-
-
-
-
-
Key Messages
-
-
Start by introducing the topic of your work and the questions you seek to answer with the numbers that follow. To help set the scene for your statistics, begin with a topic sentence that introduces the variables and the W’s (when, where and what).
-
Examples (NB. Fictional):
-
Poor: (No introductory sentence) “In 2020, there were 11,000 gun-related homicides (Figure 1)”
-
This jumps directly to presenting the data without orientating the reader to the topic and objectives.
-
Better: “What factors explain the observed rise and fall in overall homicides in England in the 2000s?”
-
This uses a rhetorical question to introduce the context (where and when) and the pattern to be investigated (the time trend). However, this does not specify the possible explanatory factors.
-
Best: “Was the substantial rise and fall in the number of homicides in the 2000s in England (Figure 1) observed across all age groups and types of weapons (Figure 2)?” There have been no numbers presented yet, just a statement that establishes the purpose of the statistics. Introducing your topic is important especially when presenting a series of charts or tables.
-
Below is a session by Robert Cuffe from the BBC talking to DfE statisticians about writing our publications and highlighting key messages in a way they can be used by journalists and the media.
-
-
-
-
-
-
Headline sections
-
It is common to use bullet points to draw out key headline messages, either for policy lines or for general interest. Here are some top tips for writing headlines:
-
-
Ask yourself if this is the most important, useful and relevant point to make? Why? What is new?
-
We recommend a maximum of six top headlines
-
You shouldn’t be trying to summarise all the findings in the publication
-
For regular publications, headlines won’t necessarily be the same every time
-
Headlines should be a single sentence making a single point, and be able to stand alone from the publication
-
Headlines should make sense to everyone and anyone (no jargon)
-
Structure headlines as: what has happened; why is this important - don’t give numbers without context
-
At least of your headlines should put the latest figures in the context of the longer-term change
-
Round figures in headlines, you don’t need lots of decimal places
-
If there is essential context for the headline facts then put this here
Throughout the commentary, active sub-headings outline the main messages making them memorable for users. They should be a short summary of the contents - aim for 10 words or less - rather than a full statistical headline. Avoid using exaggerated or sensationalist language that you would not use in your main commentary such as “highest ever” or “only”. Be neutral in selecting the message to use in the heading rather than favouring positive or negative messages.
-
-
-
-
Accordion section content
-
You should split your release into sections that each focus on one or two key messages, with a recommended maximum of 10 sections in the release. The whole release should take no more than 10 minutes to read. Our analytics app contains insights on how long it takes the average user to read your release.
-
To keep the release short only include information if there is something interesting to say - the commentary is there to tell a story, people looking for specific figures will use the table tool, or download the underlying data instead. Do not try to summarise every number in the commentary.
-
Avoid having large blocks of text as they are hard to read and users scan them and miss the detail. Graphs and tables break up the content but only include these where they add value; you do not need a graph or chart in every section.
-
Use plain English and shorter sentences, aim for an average of 15-20 words per sentence. Do not overload sentences with numbers and avoid ‘elevator commentary’ that describes small movements in the whole series without giving any insight (use a summary table instead if it is interesting, or leave it out entirely). Be impartial and objective; avoid using sensationalist terms or terms that reflect a judgement such as “very few” “only” or “strong”.
-
Explain complex concepts in plain English within the text. Remember that for many of our users, confidence intervals and significant differences are complex concepts that need explaining.
-
-
Do not use footnotes in the text of your content. They’re designed for reference in print, not web pages. Always consider the user need first. If the information in the footnotes is important, include it in the body text. If it’s not, leave it out.
-
-
-
-
-
Writing about characteristics
-
There is a wide range of guidance available from the GSS, ONS and the Cabinet Office around writing about characteristics. Statistics content published on EES should adhere to the principles outlined by the above. The data harmonisation champions group are in the process of collating the most recent guidance from all these sources and summarising it below.
-
If you need some steer on how to report on a particular characteristics, the below links provide some useful starting points:
For the official names of ethnicity filters to use in data files, please check our guidance on creating statistics. The below outlines some key points in writing about ethnicity in publication content.
-
-
-
Ethnic minorities and not BAME
-
-
As outlined in the CRED report, grouping ethnicity at a higher level than the 5 major ethnic groups should be avoided as it risks masking the true disparities between, and/or life experiences of, people from different ethnic groups.
-
There will, however, be specific cases where data quality or comparisons with past metrics does not allow for statistics to be produced at the level of the major or minor ethnic groups. In these cases, the group ‘ethnic minorities’ should be used exclusively to refer to all ethnic groups except the White British group, this includes White minorities, such as Gypsy, Roma and Irish Traveller groups.
-
For comparisons with the White group as a whole, ‘all other ethnic groups combined’ or ‘ethnic minorities (excluding White minorities)’ should be used. It may also be necessary to refer to ‘White’ and ‘other than White’ if space is limited, but this should be avoided if possible.
-
Do not use the terms ‘non-White’, BAME (Black, Asian, and Minority Ethnic), or BME (Black and minority ethnic). These terms can be perceived as demeaning and can be confusing as to what groups are being included and excluded. For comparison with past metrics, BAME and BME should be replaced by the term “ethnic minorities (excluding White minorities)” unless White minorities were specifically mentioned.
-
-
-
-
Ordering of groups
-
-
There are several methods of ordering the ethnic groups, depending on where and how they are used:
-
-
Alphabetical: use in tables and when listing ethnic groups (with ‘Other’ and sometimes ‘Unknown’ as a final category)
-
In expected order of size (with largest first): useful in charts and visualisations as it makes data and patterns easier to read
-
-
-
-
-
-
-
Reporting and interpreting numbers
-
Reporting the numbers you work with is an important first step toward writing effective numeric descriptions. By including numbers in text, table or chart, you give the user the raw materials with which to perform comparisons across time, places or groups. However, if you stop there, you leave it to your readers to figure out how those data answer the question at hand.
-
Poor: “In 2010, there were 20, 370 overall homicides related to crime, 13,000 which were related to gun incidents, 7,370 related to other weapons. In 2020, they were 18,900 overall homicides, 11,000 which were related to gun incidents, 9,900 related to other weapons (Figure 1)”.
-
The description above simply lists statistics from charts without explaining how they relate to one another or how the statistics address the initial question in the opening paragraph.
-
Better: “The total number of homicides rose until the mid-2000s and then declined until 2020. As shown in Figure 1, the increase and subsequent decrease in homicides were driven by trends in gun-related homicides. In 2020, there was roughly 1.5 times as many homicides were committed with guns as with other types of weapons (11,000 versus 7265; Figure 1); whereas in 2010, roughly 2 times as many homicides were committed with guns versus other weapons, 13000 and 6500, respectively. Figure 2 examines whether gun-related homicides showed the same time trend among all age groups. As shown in Figure 2 in the two youngest groups of offenders, gun-related homicides increased substantially between 2000 and 2010, and then decreased steadily until 2020. In contrast, the number of gun related homicides committed by older offenders decreased slowly throughout the time-period shown”.
-
Try to use prose to summarise the patterns so your user can see the general relationship in the table or chart – the forest not the individual trees. Try not to report every number from the table or chart or pick a few arbitrary numbers to contrast in sentence form without considering whether or not those numbers represent an underlying general pattern. Paint the big picture rather than reiterating all the little details. This will help you tell a clear story with numbers as evidence.
-
-
-
-
Creating hyperlinks
-
Avoid using full URLs in text. Hyperlinks should be used and they should provide a clear description of the destination. Avoid using ‘For more information click here’. Screen readers often collate all links on a page into one list, so having numerous ‘click here’ links listed is confusing to the user and gives no description of the destinations. ‘For more information see Guidance to support the summer 2021 exams’ is an example of a good hyperlink.
-
Best practice on creating hyperlinks, particularly how to name them and common pitfalls to avoid can be found on this introduction to html guide. Those of us using EES don’t need to worry about writing the raw html for the anchor links, and should instead focus on the section referring to how to name and title links.
-
-
-
-
Search engine optimisation
-
Search engine optimisation (SEO) makes it easier for users to find your data through search engines like Google. Some top tips include:
-
-
Keeping your release title shorter than 50-60 characters. This means the full title can be displayed on the search engine results
-
Avoid listing key words: search engines penalise anything not recognised as a full sentence.
-
Make use of our analytics app to explore what your users are doing: what accordions are they clicking on, what are they searching for? This could give an idea of what content you should focus on in future, and which areas are no longer of interest to most of your users.
-
-
Following best practice in writing about statistics is of increasing importance. As shown in the below example any sentence could be pulled out into a snippet and shown in a search engine to users who are searching for related information:
-
-
We should make a concerted effort to ensure that we are answering the questions people are interested in as search engines are getting smarter and pulling this information directly out of webpages. See the following example of a google search using a snippet from one of our publications as an answer in the search engine results itself:
-
-
-
-
-
-
Data visualisation
-
-
EES allows you to build charts and tables to visually represent information from your release, this can be particularly helpful to show trends over time, across geographies or highlight interesting stories in the data.
-
-
-
Visualisation titles
-
For tables and charts, active titles are descriptive and tell the trend by highlighting the main story. They should be short - aim for 10 words or less and avoid going over more than one line. Active titles give users the main message without having to find the text that accompanies the chart, and makes it easier for journalists to use your chart directly without having to write their own summary.
-
Put information such as the measure, source, population, geographical coverage and time period in a caption if they are not obvious from the chart content. Add further context and information in the main text. Remember that tables and charts should be usable even if isolated from the rest of the release.
-
-
-
-
Using data blocks effectively
-
Aim for fewer tables, and keep them small. As a guide, we suggest no more than one table per accordion section.
-
Presentation tables are the tables you include within the accordion sections of your release to quickly visualise numbers. Unlike the underlying data files, the presentation tables focus on specific parts of the story you are telling. They are distinct from, and should never be a copy of an underlying data file.
-
You should only be including small tables sparingly where they add value to your release and not use them as a straight copy of the ready-made Excel tables previously published on gov.uk. The data itself is there for users to access through other means, therefore any data in the commentary should only be there if it is enhancing the story.
-
-
You do not need to recreate all of the old excel tables, users can find the numbers they are interested in using the table tool, or analyse the underlying data if they want the data behind the release.
-
-
-
-
-
Charts in EES
-
Any charts in your release should be appropriate for the data, reinforcing the key messages. For more detailed information on how to build charts in EES, visit our EES charts guidance section.
-
Any analyst building charts in or outside of EES should make themselves familiar with the [GSS data visualisation guidance](GSS colours in visualisations guidance.
-
More complex charts cannot currently be built in EES. You should only use complex charts where there is a clear user need, as simple charts are the easiest for users to understand. Please see the ONS guidance for visualisations or contact the Statstics Development Team if you’re considering a more complex chart built outside of EES.
-
DWP have created a Data Visualisation Thinking course that may be useful to look at when creating more complex charts.
-
-
-
-
Colours in charts
-
The most important consideration when using colour is to avoid relying on it for interpretation. It should be seen as an enhancement, and your charts should be understandable without it.
-
For any charts made outside of EES you should use the suggested colours from the GSS colours in visualisations guidance. The following sections show the colours that are available for charts in EES and suggest the way to use them with one another, more detail on why can be found in the GSS guidance itself.
-
We strongly recommend you stick to the codes as outlined, if you can’t for any reason, please make sure you follow the guidance on developing your own colour palette and get in touch with us so we can understand why.
-
-
-
Categorical colour palette
-
-
Categorical data can be divided into groups or categories by using names or labels. This palette has four colours. We recommend a limit of four categories as best practice for basic data visualisations. The ordering of the palette is important as not all colours are accessible when paired together.
-
-
-
-
Colour name
-
Hex code
-
RGB
-
CYMK
-
-
-
-
-
Dark blue
-
#12436D
-
18, 67, 109
-
36, 16, 0, 57
-
-
-
Turquoise
-
#28A197
-
40, 161, 151
-
75, 0, 6, 37
-
-
-
Dark pink
-
#801650
-
128, 22, 80
-
0, 83, 38, 50
-
-
-
Orange
-
#F46A25
-
244, 106, 37
-
0, 57, 85, 4
-
-
-
-
-
-
-
Sequential colour palette
-
-
Sequential data is any sort of data where the order of series has some meaning. For example, age groups ascending in age. This palette has three colours and even this pushes the boundaries for contrast ratios. Any charts made with shades of a specific hue should be accessible without colour.
-
-
-
-
Colour name
-
Hex code
-
RGB
-
CYMK
-
-
-
-
-
Dark blue
-
#12436D
-
18, 67, 109
-
36, 16, 0, 57
-
-
-
Mid blue
-
#2073BC
-
32, 115, 188
-
0, 57, 85, 4
-
-
-
Light blue
-
#6BACE6
-
107, 172, 230
-
75, 0, 6, 37
-
-
-
-
-
-
-
Focus palette
-
-
On focus charts, colour is used to highlight specific elements to help users understand the information. One line out of many will be highlighted as a colour and the rest of the series remain grey.
-
-
-
-
Colour name
-
Hex code
-
RGB
-
CYMK
-
-
-
-
-
Dark blue
-
#12436D
-
18, 67, 109
-
36, 16, 0, 57
-
-
-
Grey
-
#BFBFBF
-
191, 191, 191
-
0, 0, 0, 25
-
-
-
-
-
-
-
Larger palettes
-
-
All charts should be made as simple as possible so that the message is easy to understand and interpret.
-
If you do want to use additional colours you need to ensure that the chart is understandable without colour as no series with more than four colours will ever be fully accessible alone.
-
For categorical data, we recommend the following extensions to the categorical palette. Once again, the ordering is important. If you’re making a chart in EES these are not default colours, so you will need to use the colour picker to specify them.
-
-
-
-
Colour name
-
Hex code
-
RGB
-
CYMK
-
-
-
-
-
Dark blue
-
#12436D
-
18, 67, 109
-
36, 16, 0, 57
-
-
-
Turquoise
-
#28A197
-
40, 161, 151
-
75, 0, 6, 37
-
-
-
Dark pink
-
#801650
-
128, 22, 80
-
0, 83, 38, 50
-
-
-
Orange
-
#F46A25
-
244, 106, 37
-
0, 57, 85, 4
-
-
-
Dark grey
-
#3D3D3D
-
61, 61, 61
-
0, 0, 0, 76
-
-
-
Light purple
-
#A285D1
-
162, 133, 209
-
22, 36, 0, 18
-
-
-
-
For larger palettes when using sequential data, we recommend using ColorBrewer to find the hex codes for multiple shades of a given hue.
-
-
-
-
-
Visualising breaks in a timeseries
-
We recommend including any missing data from breaks in a time series in your data file using the appropriate GSS symbol, such as in this example table:
-
-
-
-
-
-
-
-
-
-
-
-
-
2013/14
-
2014/15
-
2015/16
-
2016/17
-
2017/18
-
-
-
-
-
Number of pupil enrollments
-
3,627,985
-
3,713,774
-
3,796,146
-
x
-
3,885,774
-
-
-
Number of schools
-
16,705
-
16,723
-
16,736
-
x
-
16,739
-
-
-
-
-
There may be times when including missing data increases the file size too much, or becomes unwieldy, if you’re unsure and would like advice on your data contact statistics.development@education.gov.uk.
-
-
By including the missing data in your open data files you can then create charts in EES that represent this. Start off by creating a data block with the data you want to build the chart from.
-
-
On the chart configuration tab there is a toggle for visualising non-numeric values in the data as 0. On the first chart configuration tab, there’s a check box that will toggle between showing and hiding them.
-
-
When selected, you will then see that this data appears in the chart as if the indicator values are 0.
-
-
-
-
-
-
Writing methodology
-
-
Appropriate methodological information should be made available for all published Official Statistics releases.
-
You are able to create these methodology documents within the Explore Education Statistics platform and then can attach them to relevant releases.
-
It is recommended that you write your methodology document as a .html page on the platform. If you do not have time to do this, you can either contact us to help you with that, or you can upload a PDF version. If you are uploading a pdf version then you need to make sure this is accessible.
-
For releases that have a .html methodology page the relevant link will display as an accordion section under the “help and support” section.
Once definitions have been added, you can add them as URL links into your commentary. E.g. to see the definition for ‘Otherwise vulnerable’, you’d make that phrase a hyperlink, and use https://explore-education-statistics.service.gov.uk/glossary#otherwise-vulnerable as the link.
-
Before creating a new definition, check the glossary to see if one is already there. When creating glossary entries you should use standard DfE definitions, only write your own if it is something that relates specifically to your statistics. Do not use technical terms in your definition, stick to plain English.
-
Include essential definitions in the commentary where they are relevant, you should not expect users to go to the glossary or about these statistics section for information they need to understand the statistics. Do not use the glossary to explain methodology such as classification decisions, explain these in the text if essential or otherwise in the methodology document.
-
-
-
-
-
Advice and support
-
-
There are no hard rules to follow on content design and the layout of individual statistics releases will vary. Publication teams have the power to use the above guidance to ensure your statistics are as accessible and understandable as possible.
-
We’re all well-practiced at breaking complex results down into simple messages whether it’s for inducting new starters, providing briefing for new Ministers, or explaining our work for friends and family - we just need to apply the same standards to our headline reporting.
-
For those looking for more targeted advice and support there’s a DfE Content Design Champions Group and we’re open to offering direct feedback and scrum sessions to review publications if there’s sufficient demand. Contact the Statistics Development Team if you’re interested in these or would like any advice.
-
More detailed advice and guidance on content design best practice is available from a number of resources across Government and beyond:
This link provides details on how to get readability scores on your drafting in Word.
-
Similarly, the Hemingway app is an online tool to assess readability. Whilst we might not want to use this directly for final drafting, it is useful to highlight areas of particular complexity in our current narrative. Additionally, the GSS Good Practice team set an exercise to try to write about your results without using any numbers: this can be really helpful to help understand the story you’re trying to tell with your statistics.
-
As an illustrative example of work in other Departments - DWP’s Fraud & Error statistics before (128 page release), and after their own modernisation work are worth a look.
-
-
-
-
Publication scrums
-
The Statistics Services Unit organise regular publication scrums, which enable publications to be looked at with fresh eyes through a workshop with volunteers and the publication team. The content checklist, which breaks down the components of producing Best Practice content into smaller chunks with prompt questions is used (Download content checklist), along with workshop slides.
-
If you are responsible for signing off publications, then please download and see the Statistics Leadership Group paper highlighting top tips for content design.
-
One of the early outcomes from running publication scrums, was that great content design is more of an art form than science and the scrum process was found to be really useful in enabling a wider element of scrutiny and friendly challenge, in an enjoyable environment to consider publication content.
-
You can experience how a scrum runs through watching this scrum-along (features the scrum up until groups breakout to discuss different elements) and supporting slides
-
-
-
-
A full before / after scrum along is available to watch, with slides available to download separately.
Finally, an example of a team who has been through the scrum process for 3 publications talk through their realised benefits, showcasing the type of benefits potentially others could also realise is also available to watch on sharepoint.
-
If you’re interested in being involved in future scrums, please get in contact us.
The aim is to apply a consistent, logical, structure to all data files so that they are easier to use and analyse, minimising the time spent cleaning the data. Adopting these principles will give us more power to serve the needs of the users, saving both us and them time when producing and using the data, as well as opening up further opportunities for linking data.
-
Our publication data is released under the terms of the Open Government License and by following these standards will meet at least 3 stars for Open Data.
-
-
-
How to check against these standards
-
An interactive data screener has been developed in R Shiny to automate checks against the standards as a final stage of automated quality assurance before upload to EES.
-
This can be run on any data file, though requires the EES metadata to be able to process the file. The app runs on our rsconnect servers, and is only available when using DfE kit. The app is mostly self-explanatory, though if you have any questions about it, or are curious to know more about how it works, the code is available on GitHub, and you can get in touch with us at statistics.development@education.gov.uk.
-
-
All data and EES metadata files must be run through the screening app before uploading to EES.
-
-
-
-
“Tidy datasets are all alike but every messy dataset is messy in its own way.” – Hadley Wickham
-
-
-
-
-
Overview of EES data files
-
For data to be used with the table tool and charts in EES, it needs to be provided to the following overall specifications:
The data should be layed out in line with tidy data principles, consisting of filters (categories) and indicators (values).
-
The file should contain the necessary mandatory columns (i.e. time_period, time_identifier, geographical_level, country_code and country_name);
-
The data file should have an accompanying meta-data csv file, which contains information on the nature of the columns in the data file:
-
-
whether a given column is a filter or indicator;
-
human readable name for use in tables and charts on EES;
-
filter grouping information;
-
number of decimal places to display for indicator fields (i.e. allowing a lower precision to be presented in the dervied tables than the underlying data - useful for minimising rounding errors in aggregates);
-
units for indicator fields (e.g. £, %);
-
-
The data should use the appropriate GSS codes for suppressed, low, not available and not applicable entries.
-
-
An example pair of data and meta-data files are illustrated in the files and tables below.
Note that the mandatory columns time_identifier, geographic_level and country_code are abridged in the table below to help with displaying in a web page, but are shown in the example file at the link above.
Note that for the percent_children column, the underlying data is provided to 3 d.p., but the meta data constrains it to 1 d.p. This means that figures in tables in the publication will be presented to 1 d.p., but users will have access to the higher accuracy in the underlying data. As well as allowing EES to meet different users’ needs, this also helps lower the risk of rounding errors in the underlying data creating unwanted behaviour in charts in EES.
-
-
Further information on all of the requirements for appropriately prepared data files follow in the sections below.
-
Once you have prepared a draft data file, you should always run the file through our EES data file screener. This will check for common issues that may prevent the file from being used appropriately by EES. Note that some issues may not prevent your file from uploading to EES, but would still cause undesired behaviour once on the platform, so it is imperative to screen data files before uploading.
-
-
-
-
-
-
General requirements
-
-
When publishing statistics, you should be following these standards for underlying data files.
-
For publishing on EES specifically, please note the following points:
-
-
Data files uploaded to Explore Education Statistics will be downloadable, everything in your file must be publishable.
-
You must remember to apply all of your usual suppression policies.
-
The data files run the table and chart tools, so all data that you wish to create a table or chart with must be included.
-
We expect that multiple data files may be provided for each release.
-
Keep in mind that a data block can only have a single data source. However, many blocks can share one source.
-
You should include all years of the data you available in your data file to facilitate time series analysis.
-
Each data file must be accompanied by a corresponding .csv metadata file as outlined under EES Metadata.
-
-
-
In light of recent accessibility legislation, you should aim to include at least 3 years of data in EES where it exists.
-
-
-
-
Tidy data structure
-
The move to the new platform provided the ideal opportunity to standardise and ‘tidy’ up our data to give users the consistency that they ask for. Our standards draw upon the ideas of tidy data - this means applying a logical structure to datasets so they are easier to use and analyse, minimising the time spent cleaning the data before use.
-
Here is a quick summary video of what exactly tidy data is.
-
-
-
-
-
Further details on tidy data, can be found by reading Hadley Wickham’s academic paper on Tidy Data. The key principles to remember are:
-
-
Each variable forms a separate column.
-
Each observation forms a separate row.
-
-
The variables (columns) in each of the uploaded data files will fall in to the following two categories: filters and indicators.
-
-
-
Introduction to indicators
-
-
Indicators are the measureables in any data set. They should be grouped based on the type of measurement, with a column for each different type of measurement. For example, Number of pupils and Percentage of pupils would be two disticnt indicator columns in a data file.
-
More details on indicators in the EES context are provided in the Indicators section below.
-
-
-
-
Introduction to filters
-
-
A single filter column should contain all the possible filter values for a single data-sub-aggregation. For example, many publications would have an ethnicity_major column containing all the major ethnic breakdowns contained in the data or an fsm column containing the entries FSM and non-FSM.
-
In general, analysts should use a separate column for each filter in accordance with tidy data principles. This is especially the case where data are presented for combinations of filters (i.e. cross tabulations). User testing has shown this to be the most effective way to structure data for the best user experience with the table tool.
-
-
-
-
…
-
FSM
-
Sex
-
number_pupils
-
-
-
-
-
…
-
Total
-
Total
-
1209
-
-
-
…
-
Total
-
Female
-
567
-
-
-
…
-
Total
-
Male
-
642
-
-
-
…
-
FSM
-
Total
-
406
-
-
-
…
-
FSM
-
Female
-
203
-
-
-
…
-
FSM
-
Male
-
203
-
-
-
…
-
non-FSM
-
Total
-
803
-
-
-
…
-
non-FSM
-
Female
-
364
-
-
-
…
-
non-FSM
-
Male
-
439
-
-
-
-
Where data is broken down across combinations of different filters, teams should aim to “complete the matrix”. This means that, for the given filters, all possible filter combinations should have a corresponding data entry. In this way, teams can prevent users getting the ambiguous “No data” result from EES and can instead provide more explicit codes for any missing data (e.g. not applicable, not available, suppressed, etc)
-
A possible exception to the above structure is where no filter combinations/cross-tabulations are present in a given data file. For example, this may be the case if a publication requires a highlights level table that shows a result across breakdowns of sex (Male, Female, etc) and Free School Meal status (FSM, non-FSM), but not combinations of the two (Female and FSM, Male and FSM, Female and non-FSM and Male and non-FSM). In this case, analysts may choose to use a overarching collated filter columns named breakdown_topic and breakdown as follows:
-
-
-
-
…
-
breakdown_topic
-
breakdown
-
number_pupils
-
-
-
-
-
…
-
Total
-
Total
-
1209
-
-
-
…
-
Sex
-
Female
-
567
-
-
-
…
-
Sex
-
Male
-
642
-
-
-
…
-
FSM
-
FSM
-
406
-
-
-
…
-
FSM
-
non-FSM
-
803
-
-
-
-
With the above structure, breakdown_topic should be added as the fitler_grouping_column for breakdown in the associated meta data file (and therefore should not have its own row in the meta data).
-
To re-iterate, teams should not use the above format when filter combinations/cross-tabulations are present in the data file they are producing, so breakdown_topic and breakdown should not contain entries such as “Sex and FSM” or “Female and FSM” respectively.
-
Filters come in two types: standard filters and additional filters.
-
The standard filters encompass time and geography elements (e.g. time_period, geographic_level, la_code). Specific combinations of these standard filters must be present in your data files and the contents of these filters are required to meet specific standards in order for a data file to be compatible with the table tool in EES.
-
Additional filters are the release specific characteristics that we filter our data on, e.g. school types, learner characteristics, grade thresholds, etc. Some of these filters have recommended column names and entries in order to support consistency in data files across publications. For example ethnicities should have column names of ethnicity_major, ethnicity_minor or minority_ethnic and contents should be limited to the GSS standards. Such guidelines are outlined in the Common harmonised variables section of this page, whilst further information on fiters in the EES context is given in the Filters section below.
-
-
-
-
Optimising filter-indicator combinations
-
-
The policy of creating tidy data files effectively means optimising your filter-indicator combinations for use within the EES user interface. By doing so, end users will be better able to interact with your data and find the information that they’re looking for.
-
The video below provides some context around this and shows how tidy data structures work better in the EES table tool when compared to wide (or pivoted) data sets.
-
-
-
-
The number of indicators should be kept to a minimum, whilst maintaining different types of measurements as distinct indicators. For example a wide structure might consist of something like the following:
-
-
-
-
-
-
-
-
-
-
-
-
…
-
number_pupils_passing_95
-
number_pupils_passing_94
-
percentage_pupils_passing_95
-
percentage_pupils_passing_94
-
-
-
-
-
…
-
567
-
642
-
45.7
-
51.8
-
-
-
-
-
Creating a tidy form of this data would look something more like this:
-
-
-
-
-
-
-
-
-
-
…
-
grade_range
-
number_pupils
-
percentage_pupils
-
-
-
-
-
…
-
9 to 5
-
567
-
45.7
-
-
-
…
-
9 to 4
-
642
-
51.8
-
-
-
-
This is a simplified example and your data will likely be more complex, but in making this type of change, you may be able to better identify more optimal ways of organising your data. For example, if you find that restructuring like this creates a lot of empty cells, it may be that the data has incompatible filters and can be separated out into multiple data files.
-
The next video illustrates the differences between wide and tidy data, showing examples of the same data organised in each structure.
-
-
-
-
If your current processes produce wide data that you need to switch to a tidy structure, one of doing this is using the pivot_longer() function in R. The following video demonstrates how to do that using the data shown in the previous videos.
-
-
-
-
-
-
-
-
Data format
-
These standards give you the power to format the data in a way that best meets the needs of the users. There are only a handful of formatting standards to follow to ensure best practice and consistency across all of our data.
-
-
Data files must in comma separated values (.csv) format, and use UTF-8 encoding. You can specify this when saving the file in Excel, or exporting from elsewhere.
-
-
If you need to use commas within a cell, then you must add a text delimiter such as quotes to your file to define each cell - this is often done automatically for you, though if you’re unsure then you can open up your csv file in a text editor such as notepad to check.
-
You should also ensure that your data follows the GSS Standards on symbols, though be aware to ignore the ask that symbols are included in separate cells from the data, which is unpractical and unrealistic.
-
-
-
File names
-
-
File names should only include numbers, letters, underscores or hyphens. Special characters must be avoided; for example, the following characters \ / : * ? " < > | [ ] & $ , . + are all considered special characters and are used for specific tasks in an electronic environment, which can lead to confusion in some systems. The use of non-English language letters such as á, í, ñ, è, and õ, should also be avoided.
-
File name should ideally be no more than 35-50 characters, file names that try to give too much information end up having the reverse affect and users skim over and get less value than from a more concise name. File names that extend beyond 200 characters will likely cause issues for users using the files in other programs.
-
The metadata file should have exactly the same name as the data file, with a suffix of ‘.meta’. E.g. mydatafile.csv and mydatafile.meta.csv.
-
You should avoid references to time periods in the file name as this information is shown elsewhere and this can make it harder for users to make use of the newer versions of files in future years. File names should be recyclable year on year.
-
In general you should avoid including the geographic level in the file name, unless it is a file that is specifically different (e.g. a file for school level data only).
-
-
For use with EES all file names should be in lower case and avoid special characters or spaces. Any upper case characters in file names will be forced to lower case by EES, and will appear as lower case to the users.
-
-
-
-
-
Variable names
-
-
Variable names must be in the first row of your file as the header row, must not include spaces, and ideally be formatted in snake_case for ease of use.
-
Avoid starting variable names with a numeric character.
-
As with file names, you should avoid any special characters; for example, the following characters \ / : * ? " < > | [ ] & $ , . + are all considered special characters and are used for specific tasks in an electronic environment, which can lead to confusion in some systems. The use of non-English language letters such as á, í, ñ, è, and õ, should also be avoided.
-
Variable names should ideally be kept below 25-35 characters as long names are often cut off when viewing the data file and generally fail to get the information required across to users. It is a balance between giving enough information so it’s clear what it refers to and giving so much that it’s unhelpful. Remember to make use of your public data guidance and methodology for expanding on details.
-
-
-
-
How to export data with UTF-8 encoding
-
-
Most of the time our data is exported as a .csv file it will have UTF-8 encoding by default. However, there are times when this isn’t the case, and therefore we’ll quickly run through how to check this below in each of Excel, SQL, and R.
-
Excel
-
Excel tends to save all .csv files as UTF-8, however this is not always the case, particularly if there are symbols in the file (such as £). To ensure that it saves with UTF-8 encoding you can select the following when saving a file:
-
File > Save As > CSV UTF-8 (Comma delimited) (*.csv)
-
SQL
-
For saving results out of SQL as a .csv file there isn’t an option to specify the encoding, therefore the best bet is to either open the file in Excel and specify that as above, or to run your SQL query/read your data into R and follow the guidance below.
-
R
-
When writing .csv files out of R, you’ll mostly likely be using either write.csv() from base R, or write_csv() from the readr package. For the first one, you can specify encoding using encoding = like the following example:
For write_csv(), which some of you may be using for increased processing speed, the function automatically encodes as UTF-8 format, meaning that you don’t have to do anything different!
-
-
-
-
-
How much data to publish
-
You should publish as many years of data that you have and is practicable.
-
If you are not providing a full timeseries for any reason, you must link to the older published data from your publication release page, and make sure that it’s omission is explained in your methodology and metadata documents.
-
-
-
-
Deciding what should be in a file
-
Explore Education Statistics is designed to give production teams the freedom of controlling what data users can access, and how they access it. It is expected that most releases on the platform will have multiple data files, and teams have control over how they break these files up.
-
The first key consideration is that the table tool will only create tables from a single a data file, and cannot use multiple files as sources. Therefore any data that you want to compare within a single table must in the same data file. The table tool itself is there to allow users to narrow down the amount of data they have to absorb and to be able to efficiently take away key statistics.
-
A useful way to judge how to break up data files is to consider whether all of the data in the file is appropriate to show side-by-side in the same table. If there are data that are conceptually different or may be confusing to compare side by side, then these should be in separate data files. Any data file uploaded to EES is usable by all users in the table tool, and users will be able to download the exact same files as you upload.
-
We generally recommend fewer large files over a larger number of smaller files. If you think you are having issues with file size please tell us so that we can investigate and work towards a solution with you.
-
-
-
-
File size
-
There are no character or size limits in a csv file and there is no size limit for EES, though the larger a file is, the longer it will take to upload and process. Also remember that the files you upload are the files that users will download, consider the software they may access to (e.g. Excel) and whether the size of your files are compatible with this.
-
Excel has a cell character limit of 32,760 and a row limit of 1,048,576. It is best to avoid exceeding these as some end users may struggle to open the file. One good way to cut the file down is to split after a certain number of years, or to separate out different geographic levels into separate files, providing school level data as a separate file for example. With the data all being in a tidy format these are then easy enough for secondary analysts to stitch back together if needed.
-
A rough guide to file size would be:
-
-
Anything under 10mb is relatively small
-
10mb to 100mb is a fairly common file size that most teams have
-
100mb to 500mb is a large file and will struggle to upload if not compressed to a zip folder.
-
500mb and over are very large, and sometimes may struggle to compress small enough to upload.
-
4gb or more in size is larger than any we have seen before and will likely need testing in the platform first.
-
-
Contact us if you have any issues, or files that might be over 1gb.
-
-
-
-
Data symbols
-
In line with the GSS guidance on symbols, special values should be replaced with symbols in the following situations:
-
-
-
-
-
-
-
-
-
-
Symbol
-
Usage
-
Example
-
Obsolete equivalents
-
-
-
-
-
z
-
When an observation is not applicable
-
No data for at gender level for boys at an all-girls school
-
-
-
-
x
-
When data is unavailable for other reasons
-
Data for an indicator is not collected in a certain region
-
:
-
-
-
c
-
Confidential data
-
Data has been suppressed
-
-
-
-
low
-
Rounds to 0, but is not 0
-
Rounding to the nearest thousand, 499 would otherwise show as 0. Only use 0 for true 0’s
-
~
-
-
-
u
-
When an observation is of low reliability
-
Data for a local authority is identified as missing returns so is removed from regional and national totals
-
-
-
-
-
If you have any other conventions you’ve used in previous publications, or a scenario that isn’t covered above, check the GSS guidance (ignoring the part around separate columns for symbols), and contact us.
-
-
-
-
-
EES metadata
-
-
Metadata in a machine readable (.csv) format must accompany datasets uploaded to the Explore Education Statistics platform to ensure that the files can be processed correctly. This data will not be seen by users and is purely for EES to be able to understand and read your data.
-
This EES metadata is different to any metadata files you may provide alongside your data for your users.
-
We only need to provide EES metadata for filters and indicators, we do not need to provide any EES metadata for the compulsory observational units (time and geography), as this has been standardised and the platform is expecting pre-defined columns and values in those fields.
-
We do not need to provide metadata for filters that only have a single level, as this data is not useful for EES, and is there for the benefit of end users downloading the files.
-
-
Note that any extra geography columns not specified in the allowable values above, or any time columns extra to time_period and time_identifier, are filters, and should be included in EES metadata if they have two or more levels.
-
-
With some files there may be more than one way to specify the metadata that will work in EES and give an accurate representation of the data itself. In these cases we recommend that publication teams test out different approaches and decide their approach based on what will best meet the needs of the users of their data.
-
-
-
Mandatory EES metadata columns
-
-
-
-
-
-
-
-
column
-
details
-
-
-
-
-
col_name
-
This must exactly match the name of the corresponding column in the dataset.
-
-
-
col_type
-
This must be either ‘Filter’ or ‘Indicator’.
-
-
-
label
-
This is the version of the column name that the users will see on the platform, therefore you must fill this in and not leave it blank. For example, pupil_headcount may be ‘Number of pupil enrolments’. You should aim to keep these short and descriptive, but have the freedom to decide what is best to do for your users.
-
-
-
indicator_grouping
-
This column gives production teams the option to add subheadings to group indicators in order to benefit the user. If this column is left blank, all indicators will be presented as one list of individual square radio boxes with no subheadings.
-
-
-
indicator_unit
-
If this column is left blank then this will be a number by default, alternatively you can use either of the following units for financial or percentage measures - “£”, “£m”, “%”, “pp”.
-
-
-
indicator_dp
-
This column allows you to set decimal place formatting for each of your indicators. If you leave it blank the platform will default to 2 d.p
-
-
-
filter_hint
-
This column gives you the option to add in a hint such as ‘Filter by school type’ for the filter to make the service easier for the users to navigate. If you leave the column blank, no hint will appear. Do not duplicate the column name here, as it will just appear twice
-
-
-
filter_grouping_column
-
This column should be blank unless you are wanting to group your filters. When you are wanting to group your filters this column should contain the exact name of the column/variable that you wish to group by. It is good practice to use the same variable name as that you are grouping, with _group appended at the end, i.e. ‘filter’ and ‘filter_group’
-
-
-
-
Note that if you are using percentage points (pp) you must include a clear explanation in your release and methodology, so that users can understand what you are referring to.
-
-
-
-
Example EES metadata
-
Each row represents a column in the data file.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
col_name
-
col_type
-
label
-
indicator_grouping
-
indicator_unit
-
indicator_dp
-
filter_hint
-
filter_grouping_column
-
-
-
-
-
gender
-
Filter
-
Gender
-
-
-
-
Filter by pupil gender
-
-
-
-
school_phase
-
Filter
-
School phase
-
-
-
-
Filter by the phase of the school
-
-
-
-
number_children
-
Indicator
-
Number of children
-
-
-
-
-
-
-
-
percent_children
-
Indicator
-
Percentage of children
-
-
%
-
1
-
-
-
-
-
-
-
-
-
The corresponding data file.
-
-
-
-
-
-
-
-
-
-
-
-
time_period
-
…
-
country_name
-
gender
-
school_phase
-
number_children
-
percent_children
-
-
-
-
-
2018
-
…
-
England
-
Male
-
Primary
-
240
-
26.7
-
-
-
2018
-
…
-
England
-
Female
-
Primary
-
200
-
22.2
-
-
-
2018
-
…
-
England
-
Total
-
Primary
-
440
-
48.9
-
-
-
2018
-
…
-
England
-
Male
-
Secondary
-
240
-
26.7
-
-
-
2018
-
…
-
England
-
Female
-
Secondary
-
220
-
24.4
-
-
-
2018
-
…
-
England
-
Total
-
Secondary
-
460
-
51.1
-
-
-
2018
-
…
-
England
-
Male
-
Total
-
480
-
53.3
-
-
-
2018
-
…
-
England
-
Female
-
Total
-
420
-
46.7
-
-
-
2018
-
…
-
England
-
Total
-
Total
-
900
-
100.0
-
-
-
-
-
-
-
-
Time and geography
-
-
Every observation, or row, in all of the provided data files will have a set of observational units based on the time period and geographic level that the data relates to. The number of these columns will differ across files depending on the number of geographic levels included in the publication.
-
Across every single dataset of official statistics produced by DfE, the following column names must be present at a minimum (example cell values):
-
-
-
-
-
-
-
-
-
-
-
time_period
-
time_identifier
-
geographic_level
-
country_code
-
country_name
-
-
-
-
-
201819
-
Academic year
-
National
-
E92000001
-
England
-
-
-
-
-
-
Time columns
-
We use the two columns, time_period and time_identifier, to generalise time across our underlying datasets. All data files must contain these. This is a important for general useability of our data, as well as being critical in driving the charts and tables in the Explore Education Statistics platform and making explicit reference to the time in which our measurements relate to. This is a compulsory element of any official statistics dataset.
-
If you think that your data can’t follow this format, please contact statistics.development@education.gov.uk with details so that we can discuss this.
-
-
time_period must contain either a four digit year, or a 6 digit year.
-
time_period must be numeric. This allows the platform to understand ranges and order periods in a logical manner.
-
six digit time_periods must represent consecutive years - e.g. 201718, not 201619.
-
If you’re referring to a single term you should use the academic year not the calendar year in the time_period column.
-
Conceptually different years cannot be mixed in the same dataset.
-
Conceptually different year breakdowns (e.g. term, quarter, month), can be mixed with a full year of the same type using a filter column.
-
-
-
-
Specific time standards
-
-
Producers should not mix different types of years in the same dataset. This is to prevent any chance of confusion for users selecting time periods with similar labels in the table tool. For example, you cannot have Academic year and Calendar year data in the same data file. You also cannot mix yearly breakdowns (e.g. full year, quarters, months, or terms) in the time identifier column. Instead, where it makes sense to mix these within a data file you should use a filter column as shown below. Note the use of ‘Total’, this is a part of the standards for filters.
-
-
-
-
time_period
-
time_identifier
-
quarter
-
-
-
-
-
201718
-
Academic year
-
Total
-
-
-
201718
-
Academic year
-
Q1
-
-
-
201718
-
Academic year
-
Q1-2
-
-
-
-
-
-
-
time_period
-
time_identifier
-
month
-
-
-
-
-
2017
-
Calendar year
-
Total
-
-
-
2017
-
Calendar year
-
July
-
-
-
-
If your row of data spans multiple years (e.g. is a cumulative sum between 2010 and 2018), the starting year should be made clear in the name of the indicator, with the year of the end of the time period listed as the time identifier. For example if you had been recording the number of enrolments in a Local authority since from the start of the 2010/11 Academic year to the end of the 2017/18 Academic year, your data would look like the example on the right.
-
-
-
-
time_period
-
time_identifier
-
starts_since_201011
-
-
-
-
-
201718
-
Academic year
-
190
-
-
-
201617
-
Academic year
-
173
-
-
-
-
-
-
-
List of allowable time values
-
-
All time_period values should be numeric only, below the number of digits (either 4 or 6) is defined per time_identifier below. Do not include dashes or slashes in six digit years.
-
You can only mix time_identifiers if they appear within the same table below. If they are in separate tables then they should not be mixed.
You must include time_period and time_identifier columns in your data files.
-
Your data must match the allowable values above.
-
Use ‘Reporting year’ if your data does not fit in other categories, i.e. collected on a specific day.
-
If you have different types of year such academic, calendar, and financial, these should be in separate files.
-
Use filters to add more detail if you have multiple time breakdowns in the same file (quarter/full year).
-
Where a measure spans multiple years, you should name the starting year, and set the time_period as the year published.
-
-
-
-
-
-
-
Geography columns
-
We publish at a number of different geography breakdowns and these vary from publication to publication. Every publication in the new platform must include the three compulsory geography columns - geographic_level, country_code and country_name in its data files. These are compulsory as the data we are producing must lie within a country boundary.
-
The geographic_level column should describe the level of data present in that row. Therefore data for a collection from a specific local authority would have ‘Local authority’ as the geographic_level, while a National aggregation would have ‘National’ as the geographic_level.
-
Teams should make sure that they are regularly checking their geography codes if they are not using a lookup from a maintained database (such as in the PDR). ONS have the Open Geography portal, which can be a useful way of checking these. There is a wealth of data on there, though Local authority boundaries can be hard to find, they can be found using the tabs at the top – Boundaries > Administrative Boundaries > Counties and Unitary Authorities.
If you have data from an unknown location, the standard is to use ‘Not available’ as name, and ‘x’ as the code/s, this clearly marks that the geographical data for that row is unavailable, and does so in a consistent way with the wider GSS. If you have a unique variation of a certain location, e.g. ‘Outside of England and unknown’, we may be able to add these in as exceptions. Locations like this would have a code of ‘z’ as no widely used code is applicable, please get in touch with us to discuss it further if you think this might apply to you.
-
We expect that all geography codes will be based on the Open Geography Portal and run checks against this for various levels in the data screener, please contact us if you have data that doesn’t match this.
-
-
Where you have data for a legacy LA that does not have a 9-digit new code, leave those cells as blank instead.
-
-
-
-
Different measures of geography
-
-
When using geographies that can be measured in multiple ways, you can achieve this by including a filter such as level_methodology in the example below to state how you have measured the geographic level. For example, at Local authority level you may have data that was measured by the residence of the pupil and the location of the school:
-
-
-
-
-
-
-
-
-
-
-
-
geographic_level
-
old_la_code
-
la_name
-
new_la_code
-
level_methodology
-
headcount
-
-
-
-
-
Local authority
-
373
-
Sheffield
-
E08000019
-
Pupil residence
-
689
-
-
-
Local authority
-
373
-
Sheffield
-
E08000019
-
School location
-
567
-
-
-
-
-
-
-
Allowable geographic levels
-
-
All rows must have country_code and country_name completed, regardless of geographic level. The additional required columns by level are shown below. You do not have to publish at every level, this is a guide that covers every level that can be published in the platform.
-
Where you have multiple geographic levels in a file, leave any not applicable columns blank for other rows. For example, region_name and region_code should be blank for national rows. Some levels do fit into a hierarchy, for example local authorities all have a region, in these cases you should include the higher level information too. So for a local authority row, the region and country columns would all be completed as well as the la columns.
-
-
-
-
-
-
-
-
-
geographic_level
-
required columns
-
notes
-
-
-
-
-
National
-
No additional columns
-
-
-
-
Regional
-
region_code, region_name
-
-
-
-
Local authority
-
old_la_code, new_la_code, la_name
-
It is usually good practice to include the Regional aggregations where possible given the direct link between Local authorities and Regions.
-
-
-
RSC region
-
rsc_region_lead_name
-
For RSC region data, we generally define them into lead RSC regions where the majority of the data is from.
Note that Trust ID is shown as Group ID on GIAS when looking at a Trust. MATs also have a ‘company number’, this can be included but is not mandatory.
-
-
-
Sponsor
-
sponsor_id, sponsor_name
-
Note that Sponsor ID is shown as Group ID on GIAS when looking at a Sponsor.
-
-
-
-
-
English devolved area is used to refer to combined authorities, mayoral combined authorities and the Greater London Authority.
-
-
Planning area, School, Provider, and Institution level data will upload as normal to EES, though will not be read into the table tool or data blocks if they are mixed in with other levels. All data, including these levels are accessible in the downloadable files for users to explore in the same format as they are uploaded.
-
For Provider, any data files that only consist of this level are able to be uploaded and will be usable in the table tool. This solution is still not ideal, though it will allow you to include additional filters at provider level. Wherever there are multiple values per provider this should be marked as a filter, and where there is a single value per provider this should be marked as an indicator.
-
For School, any files consisting of school-level data should be standalone data files (that will work in the table tool, not ancillary files) and these should all be passing the EES data screener. They should also include LA level information so that users can see this when searching.
For Planning area and Institution data, any data files that only consist of these levels should be uploaded as an ancillary file rather than a data file.
-
-
-
-
-
-
-
-
-
geographic_level
-
required columns
-
notes
-
-
-
-
-
Planning area
-
No required columns, though we recommend both planning_area_code and planning_area_name
-
-
-
-
Institution
-
No required columns, though we recommend you include institution_id and institution_name to make data matching easier
-
-
-
-
-
-
If you have a level that isn’t covered above, please contact statistics.development@education.gov.uk with details and example data.
-
-
-
-
-
Using School or Provider as a filter
-
-
Provider / School breakdowns can work as a filter column in your data file but only if it is the only filter in the file. We’d recommend using school_name or provider_name as the filter, and then including the code as well, such as Malton School (URN: 121681) or Malton School (UKPRN: 10004165). You should also include LA level information so that users can see this when searching.
-
This approach will work for basic filtering of providers and schools but it may not be performant in the table tool and the number of choices made available to users may be overwhelming (especially for schools, or data set to national level).
-
-
-
-
Unknown geographic codes
-
-
-
-
-
-
-
-
-
-
Geographical code
-
Usage
-
Example
-
-
-
-
-
z
-
When a geography code is not applicable
-
Using custom geographies like “Scotland, Wales and Northern Ireland” as a group. This has no standardised geography code so will not be applicable
-
-
-
x
-
When a geography code is unavailable
-
Data is missing, the geography is unknown so no code is available
-
-
-
-
-
-
-
-Remember
-
-
-
-
You must have the minimum expected geography columns, country_code and country_name.
-
You must also have the additional columns required for all geographies included in your file.
-
The additional columns must exactly match the names above.
-
You do not have to publish at every level in the guide, publish the selection of levels you feel appropriate.
-
You should regularly check that you are using the most up to date geography names and codes.
-
Aside from blanks, no two geographic locations should have the same code.
-
You can mix geographies in the same file, and should avoid separating files by geography unless size is an issue.
-
-
-
-
-
-
-
-
Common harmonised variables
-
-
Wherever possible variables should confirm to the DfE harmonisation strategy. Having consistent variables across publications allows users to more quickly navigate and understand our statistics, whilst also facilitating more advanced methods of interacting with our data platform such as automated data retrieval for dashboards and other secondary statistics services.
-
-
-
Ethnicity
-
-
Overview
-
-
Race is a protected characteristic under the Equality Act 2010 and can include colour, nationality, ethnic or national origins. Data collected on ethnicity is self-identified and captures a person’s feelings of cultural heritage and belonging. Everyone may have slightly different ideas and feelings about what ethnicity encompasses, race and nationality may, or may not, be a part of a persons’ self-identified ethnicity.
-
This guidance has been written by collating the most up to date advice and guidance from the following sources:
Ethnicity breakdowns should be presented within the standard field names in any data files containing ethnicity breakdowns: ethnicity_major, ethnicity_minor or ethnicity_detailed. The first two are outlined in reference to the GSS guidelines on ethnic groupings below, whilst the third is for any ethnicity fields containing finer grained breakdowns than described in ethnicity_minor below.
-
The GSS publish standards on how to collect ethnicity data based on research conducted for the UK Census. The current guidelines (shown below) were developed as part of the 2011 Census and were unchanged in the 2021 census. We aim to follow as closely as reasonable to the GSS standards and guidance. Note the use of spaces around forward-slashes, which allows us to meet accessibility standards (in particular for the use of screen readers) as well as improving automated wrapping in text and value boxes.
-
-
-
-
-
-
-
-
ethnicity_major
-
ethnicity_minor
-
-
-
-
-
White
-
English / Welsh / Scottish / Northern Irish / British
-
-
-
-
Irish
-
-
-
-
Gypsy or Irish Traveller
-
-
-
-
Any other White background
-
-
-
Mixed / Multiple ethnic groups
-
White and Black Caribbean
-
-
-
-
White and Black African
-
-
-
-
White and Asian
-
-
-
-
Any other Mixed / Multiple ethnic background
-
-
-
Asian / Asian British
-
Indian
-
-
-
-
Pakistani
-
-
-
-
Bangladeshi
-
-
-
-
Chinese
-
-
-
-
Any other Asian background
-
-
-
Black / African / Caribbean / Black British
-
African
-
-
-
-
Caribbean
-
-
-
-
Any other Black / African / Caribbean background
-
-
-
Other ethnic group
-
Arab
-
-
-
-
Any other ethnic group
-
-
-
Unknown
-
Unknown
-
-
-
-
Note that these standards are written from the perspective of creating survey questions, i.e. the data collection phase. Where teams are trying to aggregate existing categories up to match the GSS guidance, careful consideration of how and where differing systems may or may not match with the guidance is necessary.
-
We recommend two alternative methods of ordering the above ethnic groups when reporting on ethnicity statistics:
-
-
Alphabetical: use in tables and when listing ethnic groups (with ‘Other ethnic group’ and sometimes ‘Unknown’ as a final category)
-
In expected order of size (with largest first): useful in charts and visualisations as it makes data and patterns easier to read
-
-
-
-
-
Reporting on broad ethnic minorities catageories (e.g. BAME)
-
-
Published in March 2021, the report of the Commission on Race and Ethnic Disparities made a number of suggestions on how we discuss and report on race in the UK.
-
One of the major elements was around aggregation of ethnic minorities into a single over-arching group (in particular BAME):
-
-
Recommendation 24: Disaggregate the term ‘BAME’ Stop using aggregated and unhelpful terms such as ‘BAME’, to better focus on understanding disparities and outcomes for specific ethnic groups.
-
-
The reasons for this approach are two-fold:
-
-
Grouping separate ethnic minorities into a single over-arching group can mask differing trends that may be present in the underlying data for groups experiencing different pressures and environments within UK society.
-
The phrasing ‘Black and Minority Ethnic’ and ‘Black, Asian and Minority Ethnic’ give an emphasis to specific ethnic groups whilst excluding others. This could be misleading or diminish consideration of the unnamed groups within the statistics.
-
-
Based on this guidance, the following is how teams should aggregate and write about ethnic minority groups.
-
Statistics producers should avoid where possible the practice of reporting aggregates grouping together disparate ethnic groups. Instead, statistics should be reported individually for the recommended top-level groups:
-
-
White
-
Mixed / Multiple ethnic groups
-
Asian / Asian British
-
Black / African / Caribbean / Black British
-
Other ethnic group
-
-
Where there is a reasonable need to publish or report statistics for a full aggregation of ethnic minorities (such as reporting on existing historical targets), official guidance is as follows:
-
-
Use ‘ethnic minorities’ to refer to all ethnic groups except the white British group. This term includes white minorities, such as Gypsy, Roma and Irish Traveller groups. For comparisons with the white group as a whole, use ‘ethnic minorities (excluding White minorities)’.
-
-
-
-
-
-
-
Filters
-
-
A filter is a variable that we break our data down by, such as school type, pupil characteristics, or NQT status. There are no required standards for these, you can include any filters that you feel benefit the users of your data.
-
Filter names must not include spaces, and ideally be formatted in snake_case for ease of use. You can use abbreviations to keep the names short and usable, your users will be able to see what each variable name means in the data guidance. Avoid starting variable names with a numeric character.
-
Aggregates should be included for all filters where possible. There will be some situations where aggregation does not work, this will not break the platform, though may make the data less user-friendly. In general, if you can’t aggregate, e.g. if you have headcount and fte, then this is an indication that they are instead separate measures and should have separate indicator columns rather than forcing them into a filter column.
-
For aggregate rows you should refer to these as ‘Total’. When using filters such as school type or FSM eligibility this can be quite simple, as shown in the example below:
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
time_period
-
time_identifier
-
geographic_level
-
country_code
-
country_name
-
school_type
-
FSM_status
-
headcount
-
-
-
-
-
201819
-
Academic year
-
National
-
E92000001
-
England
-
Total
-
Eligible
-
590
-
-
-
201819
-
Academic year
-
National
-
E92000001
-
England
-
Primary
-
Eligible
-
280
-
-
-
201819
-
Academic year
-
National
-
E92000001
-
England
-
Secondary
-
Eligible
-
310
-
-
-
-
-
It is possible that you wish to include a filter that only has a single level, like FSM_status in the above example that only has ‘Eligible’. This is to be expected, and you do not need to unnecessarily double the size of your file by adding a ‘Total’ that duplicates what is already there. Where this is the case, do not include this filter in the EES metadata as the platform does not need to read that column into the table tool.
-
-
-
Grouping filters
-
-
For some filters, this can be more complicated. You might have a great number of different filters that you would rather group together in a two-level hierarchy. You can do this by grouping a filter by a filter group. This essentially uses another filter column to provide a higher group level. This also may be helpful for EES users when creating tables. For example, you may want to group percentages into wider rate bands.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
time_period
-
time_identifier
-
geographic_level
-
country_code
-
country_name
-
absence_band_group
-
absence_band
-
-
-
-
-
2018
-
Calendar year
-
National
-
E92000001
-
England
-
60.0 – 69.9
-
67.0 - 67.9
-
-
-
2018
-
Calendar year
-
National
-
E92000001
-
England
-
60.0 – 69.9
-
62.0 - 62.9
-
-
-
2018
-
Calendar year
-
National
-
E92000001
-
England
-
10.0 – 19.9
-
16.0 - 16.9
-
-
-
-
-
-
You can also see an example with apprenticeship data file, including the corresponding EES metadata, below:
-
-
Data file.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
time_period
-
…
-
country_name
-
appr_type
-
appr_exact
-
starters
-
drop_outs
-
drop_out_rate
-
-
-
-
-
2018
-
…
-
England
-
Hair and beauty
-
Hairdressing
-
200
-
10
-
0.05
-
-
-
2018
-
…
-
England
-
Hair and beauty
-
Beauty therapy
-
100
-
2
-
0.02
-
-
-
2018
-
…
-
England
-
Rail engineering
-
Track signalling
-
80
-
4
-
0.05
-
-
-
-
-
-
Corresponding EES metadata file.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
col_name
-
col_type
-
label
-
indicator_grouping
-
indicator_unit
-
filter_hint
-
filter_grouping_column
-
-
-
-
-
starters
-
Indicator
-
Number of learners starting an apprenticeship
-
Apprentice starts
-
-
-
-
-
-
drop_outs
-
Indicator
-
Number of learners not completing an apprenticeship
-
Apprentice drop-outs
-
-
-
-
-
-
drop_out_rate
-
Indicator
-
Percentage of pupils not completing an apprenticeship
-
Apprentice drop-outs
-
%
-
-
-
-
-
appr_exact
-
Filter
-
Type of apprenticeship
-
-
-
Select the apprenticeship types you’re interested in
-
appr_type
-
-
-
-
-
-
-
-You must not include grouping filters as separate rows in the EES metadata, they should only be referenced in the filter_grouping_column.
-
-
-
-
-
-
Dates as filters
-
-
If you are publishing data with a date assigned (e.g. daily data), you can add date in as an additional filter.
-
It is important to maintain a yyyy-mm-dd format as demonstrated below, so that EES lists dates in the correct order in the table builder.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
time_period
-
time_identifier
-
date
-
geographic_level
-
country_code
-
country_name
-
absence_band
-
-
-
-
-
2020
-
Week 50
-
2020-05-14
-
National
-
E92000001
-
England
-
67.0 - 67.9
-
-
-
2020
-
Week 50
-
2020-05-15
-
National
-
E92000001
-
England
-
62.0 - 62.9
-
-
-
2020
-
Week 50
-
2020-05-16
-
National
-
E92000001
-
England
-
16.0 - 16.9
-
-
-
-
-
R automatically outputs dates in this format, but you can also change this in excel if needed, by selecting the column to change and switching from “General” to the specified date format:
-
-
-
-
-
-
-
-
-
-
-
-
-
-Remember
-
-
-
-
All filters should have a ‘Total’ aggregation where possible.
-
If a filter has one level, you should not include it in the EES metadata.
-
Where filters do not aggregate, consider if you should have the levels as indicators instead.
-
Grouping filters is an option, but only group filters where you decide it is better for your users.
-
If a filter is used to group another, then it should not appear as a separate row in the EES metadata.
-
-
-
-
-
-
-
Indicators
-
-
The indicators are the variables showing the measurements/statistics themselves, such as the number of pupils. These can be of different formats (e.g. text, numeric), although are numeric by default. The number of indicators will vary across publications and data files.
-
-
-Every variable in your dataset should have its own column, and each column should be a single data type. E.g. do not create an indicator column called “pupils” that has both the number and percentage of pupils in it. Instead, create two separate columns for each measure.
-
-
-
As an example, the number and percentage of pupil enrolments are the indicators in this dataset:
-
-
-
-
-
-
-
-
-
-
-
-
-
time_period
-
…
-
country_name
-
school_type
-
enrolments_num
-
enrolments_pc
-
-
-
-
-
201819
-
…
-
England
-
Total
-
200
-
100
-
-
-
201819
-
…
-
England
-
Primary
-
150
-
75
-
-
-
201819
-
…
-
England
-
Secondary
-
50
-
25
-
-
-
-
-
Like filters, indicator names must not include spaces, and ideally be formatted in snake_case for ease of use. You can use abbreviations to keep the names short and usable, your users will be able to see what each variable name means in the data guidance. Avoid starting variable names with a numeric character.
-
In EES, the platform itself will format large numeric values to include commas, so you do not need to have commas for separating large numbers in your data file.
-
If you are including percentages and want to show 79% in EES, you would have the value 79 in your data file and specify % as the unit for that column in the EES metadata. As best practice we recommend that you include as much detail as possible in your underlying files when it comes to decimal places. As a default EES will round decimals to 2 d.p. in the platform, though if you would prefer to display your data to a different number of d.p. in the platform please use the indicator_dp column to specify this in the .csv metadata. Likewise with financial data, for don’t include £ in the values in the data file, leave those cells as numeric only, include it as an indicator unit instead.
-
In line with GSS best practice for communicating uncertainty, when using confidence intervals you should report the upper and lower bounds as separate columns rather than as one confidence interval column:
-
-
-
-
-
-
-
-
-
-
-
-
-
-
time_period
-
…
-
country_name
-
school_type
-
estimate
-
est_lower_ci
-
est_upper_ci
-
-
-
-
-
201819
-
…
-
England
-
Total
-
100
-
96.7
-
103.3
-
-
-
201819
-
…
-
England
-
Primary
-
50
-
48.9
-
51.1
-
-
-
201819
-
…
-
England
-
Secondary
-
50
-
49.5
-
50.5
-
-
-
-
-
-
-
Indicator grouping
-
-
Many of our publications contain a large number of indicators. To improve the experience of the user in the platform we can group these under headings as shown on the right. In the example, you can see how the different indicators have been grouped into ‘Admissions’, ‘Applications’, and ‘Preferences breakdowns’ in the EES metadata, followed by how this will appear in the table tool for users.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
col_name
-
col_type
-
label
-
indicator_grouping
-
indicator_unit
-
filter_hint
-
filter_grouping_column
-
-
-
-
-
nc_year
-
Filter
-
NC Year
-
-
-
Filter by national curriculum year
-
-
-
-
admissions
-
Indicator
-
Number of admissions
-
Admissions
-
-
-
-
-
-
applications
-
Indicator
-
Number of applications received
-
Applications
-
-
-
-
-
-
online_apps
-
Indicator
-
Number of online applications
-
Applications
-
-
-
-
-
-
online_apps_%
-
Indicator
-
Percentage of online applications
-
Applications
-
-
-
-
-
-
-
-
-
Example of how this would look in Explore Education Statistics:
-Guidance for publishing public statistics dashboards
-
-
-
-
When to use a dashboard
-
-
The primary route for publishing official statistics is via Explore Education Statistics (EES). In addition to publishing on EES, there may be times where you want to compliment official statistics with a dashboard. This may be to do a deeper piece of analysis for a specific user group, or to make use of functionality that isn’t available within the platform.
-
It is important that all dashboards should have a clear use case and reasoning for why they are required. If you are considering developing and publishing a dashboard, please get in touch with the Statistics Development Team who can advise and assist with the process, and make use of the teams channel to ask questions and share what you’re working on with other teams.
-
This guidance is aimed primarily at dashboards published to supplement Official Statistics to the wider public. See the how to publish an R shiny dashboard section below for more information on the guidance for internal only R Shiny dashboards, or public facing R Shiny dashboards via shinyapps.io.
-
This page includes specific guidance for R Shiny dashboards, for teams using PowerBI, the department has a set of PowerBI dashboard standards that can be found on the DfE’s analytical services GitHub area.
-
There is also a dashboards version of the content design checklist. This checklist runs through a number of things to think about when developing dashboards to compliment official statistics and is worth using to design and also review any dashboards you own - Dashboards checklist (.xlsx). We also have a dashboard procedure checklist in this guidance page that outlines the steps required to set up an R shiny dashboard.
-
-
-
-
Things to consider
-
-
All dashboards should have a clear use case and reasoning for why they are required. They also need to meet the latest accessibility regulations for public sector services.
-
Dashboards, along with the Official Statistics that they supplement, are a digital service that we provide, and we should ensure we follow the Service Standard. The standard covers a number of aspects that all producers should be considering when developing a dashboard, and strongly influence the guidance on this page.
-
All data must be published on EES as open CSV files so that it is accessible in one place and we are not publishing anything new via dashboards.
-
When published, dashboards can be directed to from existing EES releases using the related links at the top right of a release page, or by being included as a url anywhere in the release content.
-
The following sections cover the considerations in more detail, though when developing a dashboard you need to commit to thinking about:
-
-
User needs
-
Software - long term sustainability, knowledge and resource
-
Accessibility guidance
-
Consistent styling
-
User engagement and analytics
-
-
-
-
User needs
-
When designing a government service, always start by learning about the people who will use it. If you do not understand who they are or what they need from your service, you cannot build the right thing.
-
Understanding as much of the context as possible gives you the best chance of meeting users’ needs in a simple and cost effective way.
-
The real problem might not be the one you originally thought needed solving. Testing your assumptions early and often reduces the risk of building the wrong thing.
-
Services designed around users and their needs:
-
-
are more likely to be used
-
help more people get the right outcome for them - and so achieve their policy intent
-
cost less to operate by reducing time and money spent on resolving problems
-
-
See the service manual for more information on learning about your users and their needs.
-
-
-
-
Software choices
-
When you make a decision about technology, you’re making a significant investment. The choices you make will have a huge impact on your ability to create, iterate and operate the service in a sustainable way. We should be choosing tools and technology that let us create a high quality service in a cost effective way and minimise the cost of changing direction in future.
-
For public facing dashboards we recommended using R Shiny.
-
We feel that it best meets the service standards, it aligns with the departments RAP strategy for the use of R in in Official Statistics, and the general direction of the government analytical community.
-
R Shiny is incredibly customisable, and ideal for providing everything in one place - interactivity, user customised printable PDFs (replacing old mail merge solutions), and ease of use on a phone!
-
R Shiny takes advantage of transferable knowledge and shared resources that already exist to minimise the costs and maximise the shared benefits.
The most common dashboard options available to us are:
-
-
R Shiny
-
Python Dash (or other Python-based alternatives)
-
PowerBI
-
-
Some considerations to think about when choosing the tooling / software for a dashboard:
-
-
will it be open source? (Service standards 12 and 13)
-
what skills do your team already have, or are already developing?
-
can you reuse anything that someone else has already done?
-
do you have flexibility in formatting and styling?
-
will you be able to maintain it long term?
-
does it give the flexibility required to meet accessibility requirements?
-
what costs will it involve? (Consider learning and development time and courses as well as hosting)
-
will it allow you to develop automated testing and CI to QA our dashboards?
-
does it align with the AF expectation to move towards R and Python for analysis?
-
-
The guidance on this page will focus on R Shiny. However, if you are using PowerBI you should also make use of the department’s PowerBI Dashboard standards, and aim to make use of reusable configurations where possible to ensure that duplication of effort across the department is minimised.
-
-
-
-
Accessibility
-
Government services must work for everyone who needs to use them. Public sector organisations have a legal duty to consider everyone’s needs when they’re designing and delivering services. Making a website or mobile app accessible means making sure it can be used by as many people as possible. This includes those with:
-
-
impaired vision
-
motor difficulties
-
cognitive impairments or learning disabilities
-
deafness or impaired hearing
-
-
Accessibility means more than putting things online. It means making your content and design clear and simple enough so that most people can use it without needing to adapt it, while supporting those who do need to adapt things.
-
The accessibility regulations came into force for public sector bodies on 23 September 2018. They say you must make your website or mobile app more accessible by making it ‘perceivable, operable, understandable and robust’. The full name of the accessibility regulations is the Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018. The accessibility regulations build on your existing obligations to people who have a disability under the Equality Act 2010 (or the Disability Discrimination Act 1995 in Northern Ireland).
-
These regulations apply to all dashboards created for Official statistics. All dashboards should:
Analysts should familiarise themselves with the gov.uk guidance on service accessibility
-
-
-
-
Governance
-
All teams developing dashboards to accompany Official Statistics should contact the Statistics Development Team for advice on the level of approval required as there may be times when HoP approval is needed in addition to team/unit leader approval. Approvals should be sent to the Statistics Development Team when asking for a new dashboard to be hosted on shinyapps.io.
-
The standards for dashboards, and department strategy are maintained by the Central Statistics Standards Unit and governed by the Statistics Leadership Group, which is made up of all senior statisticians owning Official Statistics publications in the department.
-
Regular reflection on how teams are doing and finding ways to improve is an important part of good governance. We recommend that Senior Statisticians responsible for dashboards ensure that they are regularly reviewed, covering:
overall coherence with central standards and strategy found on this page
-
-
-
-
-
Iterative development
-
Using agile methods means getting your service in front of real users as soon as possible. Then observing and generating data on how they use it and iterating the service based on what you’ve learned. Because you’re not specifying everything up front before you’ve developed an understanding of what users need, agile methods reduce the risk of delivering the wrong thing.
-
Teams should aim to get a working version of their dashboard out to users as soon as possible to prototype the design and content.
-
Iteration isn’t just for the early stages of a service’s development and services are never ‘finished’. Using agile methods means getting real people using your service as early as possible. Then making improvements throughout the lifetime of the service.
-
Making improvements means more than doing basic maintenance like fixing bugs in code, deploying security patches and keeping call centre scripts up to date. If that’s all you do, you’ll be fixing symptoms rather than underlying problems. And over time, the service will stop meeting user needs.
-
Continuous improvement means you can respond to changes in user needs, technology or government policy throughout the lifetime of the service. So rather than having to be replaced, the service stays relevant until it’s ready to be retired.
-
-
-
-
Code testing
-
Testing is a way to capture desired behaviour of your code, in such a way that you can automatically verify that it keeps working the way you expect over time. It is essential for making sure that code works the way that you intend it to, and keeps working even after you make changes to the code so that your users have access to a stable service. You need to test your service regularly as part of quality assurance (QA) to make sure that it:
-
-
is easy to use for anyone who needs to use it, regardless of the device they’re using
-
is stable, secure and works quickly, regardless of how many people need to use it
-
can be iterated quickly to meet changes to user needs or the political environment
-
-
Tests can come in a variety of shapes and sizes, good starting points for analysts new to testing are the Duck Book, and DfE good code practice.
-
You should aim to automate as much of your testing as possible and run your test suite as part of continuous integration (where your tests form part of your codebase). By testing your code automatically every time you make a change, you’ll be able to find defects more quickly. Getting fast feedback means you can respond to any problems quickly and make changes when you need to. You can also spot defects before they develop into bigger problems that are more complicated and expensive to fix.
-
For dashboards created in R Shiny, we strongly recommend a mix of unit tests and UI tests (integration), run using GitHub Actions for continuous integration (CI). All applications should have an appropriate level of test coverage before being published. More information and details on how to get started with this can be found in the testing R Shiny section below.
-
-
-
-
Assessing engagement
-
Work out what success looks like for your service and identify metrics which will tell you what’s working and what can be improved, combined with user research. For dashboards this will likely be things like the number of users and interactions with what you have created.
-
Defining what “good” looks like and identifying appropriate metrics means that you’ll know whether the service is solving the problem that it’s meant to solve.
-
Collecting the right engagement data means you’ll be alerted to potential problems with your service. And when you make a change to the service, you’ll be able to tell whether it had the effect you expected. In practice this will often mean setting up analytics and feedback surveys for dashboards and monitoring the data you get back.
-
At a minumum you should be requesting feedback from users via a survey hosted on the dashboard and reviewing this on a regular basis. An example of this kind of feedback survey is the beta banner survey on Explore Education Statistics.
-
-
Google Analytics is a free service that collects information on who visits your webpage and how they interact with it. You can set up basic Google Analytics for your published dashboard in a few simple steps outlined in this article: Add Google Analytics to a Shiny app and view a more complex example in this example file. If you’re planning to publish a dashboard, or to set up Google Analytics for a published dashboard, please contact statistics.development@education.gov.uk.
-
-
-
-
Peer review
-
Peer review is a quality assurance activity, where an analyst other than the original author, views and tests the usage of a product or specific piece of code. This allows a fresh pair of eyes to take a look at your work. It validates that you have taken a correct approach and may highlight errors. This constructive feedback helps you to improve the quality. It provides confidence in your work, and ensures that it is fit for purpose.
-
-
Dashboards must always be peer reviewed within the team they are created.
-
Dashboards should also be peer reviewed by analysts outside of the subject area of the team.
-
-
The Central Statistics Unit has a number of analysts experience with R Shiny dashboards and are happy to review any dashboards created, contact statistcs.development@education.gov.uk if you’re interested in this. For more guidance on how to peer review, see the peer review section of the Duck Book.
-
-
-
-
-
How to publish an R Shiny dashboard
-
-
How you choose to publish your R Shiny application will depend on the type of access you require for it.
-
-
-
Internal only dashboards
-
It’s possible to publish dashboards that are only accessible to those on DfE kit, to do this you will need to publish via the departments RSConnect servers.
-
You will need:
-
-
A finished app, in line with the guidance on this page
-
The code to be in a Git repository in the dfe-gov-uk Azure DevOps space
-
-
To publish the app, you’ll need to set up a pipeline in Azure DevOps, guidance for how to do this can be found in the R community teams area.
-
Access to applications on RSConnect are locked down by default, once the pipeline is set up and you’ve deployed the app you’ll need to request for its access to be opened up by using an RSConnect request on ServiceNow.
-
If you’re running the app from an internal database, you’ll need to contact the database owner to set up a local login, and then store those as variables against your specific app in rsconnect. You can raise a request to do this via ServiceNow and selecting ‘Change app variables’.
-
-
-
-
Public dashboards
-
The majority of dashboards made to support and augment our Official Statistics will be public facing. For public facing shiny apps you should publish via shinyapps.io. The statistics development team manage a subscription for this and can help you get set up.
If the data underlying the dashboard is currently unpublished, you will need to create dummy data to use in GitHub until the data becomes published (see dummy data guidance section).
-
-
To set up a new app, send the above to statistics.development@education.gov.uk. If your code is not yet hosted in the dfe-analytical-services area you can request for the repository to be moved at the same time as sending approvals.
-
-
-
Dummy data
-
-
When creating a public dashboard, all code and data will be hosted in the dfe-analytical-services GitHub repo. This is a public repo, meaning anything stored in here is publicly available. Therefore if you are creating a dashboard with previously unpublished data, you should provide dummy data for the GitHub repo, and only add the real data on your publication date.
-
Your dummy data should:
-
-
Use the exact same structure and variable names as your real data
-
Use random values in place of real values (one example of how to do this is using the rnorm() function)
-
Simulate the types of suppression, missing data or anomalies seen in your real data, to ensure the dashboard can account for these.
-
-
-
-
-
Testing with unpublished data
-
-
While you must use dummy data in your GitHub repo, it is understandable that we should test the dashboard works with the real data before it goes live.
The .gitignore file is a plain text file that tells git to ignore specified files in commits and pushes to the repo. Therefore, the first step when wanting to test unpublished data in a dashboard is to add the unpublished data file names to the .gitignore file.
-
Adding the file name alone will ensure it is ignored no matter where in the project area you save it (i.e. in any folder). Once this is done, you can add your unpublished data file to your local area, run the app locally, and make edits/commits without uploading the data to Github.
-
This .GitIgnore guidance page has great guidance on how you can utilize wildcards to capture all the files you might want to ignore.
-
If you have any questions on this process please do contact us at statistics.development@education.gov.uk.
-
-
-
-
-
-
R Shiny
-
-
Shiny is an R package that makes it easy to build interactive web apps straight from R. You can host standalone apps on a webpage, embed them in R Markdown documents or build dashboards. You can also extend your Shiny apps with CSS themes, htmlwidgets, and JavaScript actions.
-
The guidance on this page assumes you have prior knowledge of R, if you are new to R, then take a look at our R resources page to get started.
-
-
-
How to start
-
There are a lot of resources already available to support you when working with R Shiny, if you’re new, then this blog post provides a gentle introduction.
-
Our template shiny app repository is a useful starting point for all public facing dashboards in the department. Please see the dashboard template section for guidance on what the dashboard template includes and how to use it.
Finally, you should seek to make use of the community that is already out there, see what others are doing, ask them questions and for advice on any decisions or problems that you’re facing, and share what it is that you’re doing. - The Statistics Development Team are experienced with R Shiny and happy to help or offer advice, and there is the DfE R community on teams. - There is a bank of tips and tricks stored in the DfE R teams area too. - Going beyond DfE there’s a wealth of resources and communities online, including Stack Overflow discussions, cross government slack channels (e.g. #R and #Shiny on govdatascience.slack.com) and even tweets about R Shiny on twitter.
-
-
-
Dashboard procedure checklist
-
-
This checklist outlines the standard procedure for teams who are wishing to produce a public R shiny dashboard.
Ask the Statistics Deveopment Team to create you a repository in the DfE analytical services area, providing the name of the dashboard and the GitHub accounts of anyone who will be contributing to the code. You should aim to have two analysts working on the code development and a line manager for review purposes. Further colleagues with review responsibilities (policy colleagues, G6 and above, etc.) can be given access to a demo-site, rather than the repository (see guidance for this below in ‘Setting up a development/demo dashboard area’).
-
Clone the repo to your device so you can develop your code. Open the repo page in GitHub, click the green ‘Code’ button, and copy the URL it provides. Then, open R Studio on your device, click file > new project > version control > git, paste the repository URL you copied from GitHub, give your local project area a name, and choose where to save it (i.e. on your computer’s C: drive, outside of the OneDrive-synced folders).
-
-
Once you’re set up, there are certain parts of the code you need to update:
-
-
In the global.R script, update all of the site URL’s and EES publication names to your own.
-
In the ui.R script, update the tags$title(), the information in the meta_general(), and the secondary text in shinyGovstyle::header().
-
Got to the .github > workflows folder, and open the deploy-shiny.yaml file. At the bottom, update the appName in rsconnect::deployApp() - this is what will appear in the URLs (i.e. must align with the links in global).
-
Update the ‘Support and feedback’ tab (in R > standard_panels.R) with your teams information. We also recommend creating a feedback form for users of your dashboard, and adding a link to that on this page.
-
Update the README.md file (you can do this easily directly in GitHub). This is the file that renders below your repo in GitHub for users to see.
-
Begin adding your own dashboard content. If you copy and paste any parts of the code (i.e. to create new tabs) you must change all of the IDs so there are no repeated IDs in the UI, otherwise the app will run with no UI elements. You should add UI and unit tests as you develop your code as a form of automated QA (see our guidance on UI tests and guidance on unit tests).
-
-
You must contact the statistics development team for the following:
-
-
To add the shinyapps.io secret and token to your GitHub repo, therefore enabling the app to be hosted on shinyapps.io.
-
To create an area for your team in Google Analytics, to track the user analytics of your dashboard.
-
-
Setting up a development/demo dashboard area:
-
-
While developing your dashboard, you may want a private, demo-version to share with policy or senior colleagues for review and feedback. This version must use either published data or dummy data and can not use unpublished data, since this cannot be uploaded to GitHub until the day of publication (see our dummy data guidance for public dashboards).
-
Ensure that prior to contacting the statistics development team, you have updated all of the URL’s and other items listed above.
-
You must contact the Statistics Deveopment Team to add the shinyapps.io secret and token to your GitHub repo, therefore enabling the app to be hosted on shinyapps.io. Once this is done you will have a browser link you can use to access the dashboard. We can make this private such that there is a list of approved viewers who must log in to view the dashboard - please provide the email addresses of any colleagues who you wish to have access to the private version during development.
-
-
You must have done the following before a dashboard can be published (the statistics development team must review and approve that these have been met):
-
-
Accessibility testing the dashboard and updating the accessibility statement. The accessibility testing guidance later in this section outlines how teams can do this.
-
You should test how your dashboard appears and performs on mobile devices. you can do this by opening your dashboard in chrome/edge, right clicking anywhere on the page and selecting ‘inspect’. you will then see a new panel on the right hand side of your screen. To see your dashboard how a mobile user would, click the mobile/tablet button (see the image below).
-
-
-
-
Setting up UI and unit tests (UI tests are a part of the automated QA process and will run via GitHub actions each time you create a pull request). See our guidance on UI tests and guidance on unit tests.
The underlying data for the dashboard must still be uploaded to EES in the tidy data format to be used in the table tool (check this using our data screener)
-
Decide where you are going to provide the public the link to your dashboard. You can easily add a link to your EES publications. If you have a draft release that is going out on the same day as the dashboard, you can add the link into your EES draft while the shinyapps.io page is still private. This is because the link will not change when the dashboard is made public, the log-in screen will simply be removed.
-
-
-
-
-
-
Standards to follow
-
We expect all dashboards to follow a minimum set of standards to ensure coherence between our products and a minimum standard of quality for our end users.
-
These standards are constantly evolving, and all feedback and contributions are welcome, contact us at statistics.development@education.gov.uk.
As a minimum we expect all dashboards to be checked using the two tools below (lighthouse and shinya11y) and have an accessibility statement before being published.
Lighthouse in Google Chrome is an easy way to quickly rate your accessibility. Open your app in Chrome, right click anywhere on the page and select “Inspect”. From there, navigate to “Lighthouse” in the top grey bar, and click “Generate report”. This generates scores for accessibility, best practices and SEO for your application or web page:
-
-
To complement this, we recommend also using the shinya11y package to look at accessibility across your dashboard. Simply install and load the package, then include use_tota11y() at the top of your UI script. This brings up an interface that helps you examine different accessibility aspects in your app, like colour contrast, alt-text and what screen readers will detect:
-
-
Automated tools can’t check for everything and there’s no substitute for giving your dashboard a manual check. You should consider what users need from your dashboard and ensure that that information is accessible to all. For example, can someone using a screen reader get the same information as you can see on the screen from a downloadable csv or from the alt text that you have provided? Everything that you can see, you should also be able to read with a screen reader. Use the Edge narrator tool to test out how your dashboard works in practice.
-
One example of a dashboard for Official statistics that meets these regulations is the SCAP LA school places scorecards app. Their accessibility statement is clearly labelled, explains what checks have been carried out, what the known limitations are and the plans in place to fix them:
-
-
-
-
-
-
Styling
-
-
-
All dashboards should have a link to the source code, and information about who to contact for feedback/issues. You should be familiar with and follow the gov.uk style guide as appropriate.
-
Teams should pick from the gov.uk design system colours when creating dashboards. The style sheet in our template repository makes use of these colours, though this can be extended if needed to suit your needs. Be careful when choosing colours, and make sure that colours have sufficient contrast to be placed next to each other to meet WCAG AA standards using the colour contrast checker.
-
Charts using ggplot2 and plotly can take custom palettes using the scale_fill_manual() function, e.g. below:
You can use in-line CSS to make these style changes, or make edits to the style sheet. Editing the style sheet is a neater solution if you want to apply changes across your app, and keeps your UI code clean. You can find out how styles are configured by running your app, right clicking and selecting “Inspect” to view the element you wish to change. You can then find these lines in the CSS stylesheet and change a number of things including fonts, font colours and background colours.
To ensure that they are reliable, dashboards should have an appropriate amount of automated tests before they can be published. We recommend using a mix of UI and unit tests, but the number and type of tests that you need to run will depend on the content of your application.
Performance profiling represents a chance to breakdown how different elements within your dashboard perform relative to each other. This consists of running through a set of actions on a local run of your dashboard and timing each one. A useful tool for performing these tests is profvis, which carries out all the timings and visualisation of the results for you, whilst all you have to do is run the app and step through the actions that you want to profile.
-
The basic steps are as follows.
-
With your dashboard repository open in RStudio, run the following commands from the R console
-
install.packages("profvis")
-
profvis(shiny::runApp())
-
Then go through a set of interactions in your dashboard (e.g. navigate to the dashboard page, step through changing the selected choices in each input field, switch between any different tabs and cycle through the relevant inputs again). Once you’ve performed a representative range of actions in the dashboard, close the window and go back to the RStudio window.
-
After several seconds (depending on the range of actions you performed), a visualisation will appear in the main RStudio editor window showing the results of the profiling.
-
The profvis results are shown in the flame profiling chart. The top half of the chart shows any processing time measurements above 10ms for each line of code. It can be useful to focus in on lines that have measured times above 10ms by selecting “Hide lines of code with zero time” from the options menu in the top right.
-
-
Lines with timings of greater than about 5-100ms may warrant further investigation to see if they warrant some optimization. Common routes to optimising code are:
-
-
avoid any data processing in server.R, e.g. avoid using the summarise() function;
-
use reactive() elements to minimise repeated calls of given functions;
-
use memoise() to cache results of more intensive functions (e.g. ggplot) to minimise repeated calls.
Whilst profvis can help identify any bottlenecks, this will ideally be complemented in the future by full load testing of dashboards, whereby the behaviour of dashboards under real-life high demand scenarios can be tested. However, this type of testing is unavailable whilst our applications are hosted on ShinyApps. We will offer support on load testing once we move dashboards on to our own server systems.
-
-
-
-
UI Tests
-
-
User interface (UI) tests should be used for all apps. These tests should check that:
-
-
Your app loads up correctly
-
Content appears as expected when you switch between tabs
-
Content updates as expected when you change inputs
-
-
The shinytest package is a really easy way to create these tests on an RShiny app. Simply load in the package and run shinytest::recordTest() to open up the testing environment to get started.
-
Each test should produce one “expected results” JSON file. You can track particular elements of your app by holding CTRL and clicking on an element in the testing window.
-
For example, the SCAP LA school places scorecards app has tests that check that the app functions as expected when different combinations of filters are selected.
-
Each test is separated out by comments, clearly stating what is being tested. - app\(setInputs()** tells the UI test to select an input in the app - **app\)snapshot() tells the UI test which outputs to capture (i.e. which outputs you’d expect to change and want to test)
-
-
The expected outputs are saved in a UI_tests-expected folder, and every time you run a UI test, the outputs will be compared to these expected snapshots.
-
An example of a basic shiny test to check that your app loads up correctly can be found in our shiny template repo.
-
-
-
-
Unit Tests
-
-
Unit tests should be used for apps that contain custom functions created from scratch, or that rely on existing functions being combined to interact in a certain way.
-
Unit testing checks, for a set of defined variables, that the output is always the same when you run your custom function/s. This ensures that if slight tweaks are made to your function, or functions within packages you are using are updated, that your custom function still works.
-
For example, our QA app has many custom functions that we create to screen files before upload to EES. We have a series of dummy files that we have created to either pass or fail certain tests. Unit tests are written to check that these files always pass or fail as expected.
-
-
-
-
Tests and deployment
-
-
Both UI and unit tests need to be added to your app’s deployment pipeline. This ensures that the app will not be published if any of the tests fail.
-
We recommend using Github Actions to deploy the latest code whenever a push to your master branch occurs - this ensures that the published version of your app stays up to date.
-
You should also use GitHub Actions to run the automated tests for your app, which we recommend are done on at least every pull request.
-
If you’ve started from our template repository then all of this will be mostly set up for you to tweak and expand on, but if you haven’t then you’ll need to add the yaml files from the .github/workflows folder to your repository.
-
GitHub actions are already well documented and their own guidance should be able to walk you through anything you need. That being said, if there’s anything else you’d like to see here let us know.
-
-
-
-
-
-
Connecting to databases
-
-
We don’t yet have a database server set up that can be accessed by public facing dashboards, though we are working to put this in place. In the meantime, there a few alternative options for storing data that the dashboard can run off.
-
-
Store the data within the repo (e.g. CSV files in a /data folder). Note that only published data should be stored in a repo. If your working on a dashboard that uses unpublished data then please see the dummy data section for guidance.
-
Use Google sheets
-
Use Dropbox
-
-
If you are running an internal-only app then you can connect to our internal SQL servers, information on how to do this is in the R community teams area.
-
-
-
-
Secure variables
-
-
See our Git page for guidance on storing secure variables in repositories.
-
-
-
-
-
Publishing your dashboard
-
-
DfE Shiny applications are published via the DfE Analytical Services shinyapps.io account. You need to alert the statistics development team of any new dashboard publication as early in development as possible and keep us updated on expected publication date. Update the stats development team on any subsequent data or major functional updates to the dashboard publication at least a week prior to re-publishing with the update. Deploying to shinyapps requires the DfE platform codes to be entered into the repository secrets area of your app. This needs to be done by the stats development team. Authorisation of a publication should be requested from the relevant G6 or DD and the stats development team (with the former authorisation e-mail being forwarded on to the Statistics Deveopment Team).
-
If you are publishing a dashboard using already published data, then all of your code and data should be on GitHub. You may have decided to password-protect the dashboard URL, in which case, you should make the Statistics Deveopment Team aware of your publication date so that they can remove the password-protection at 9:30 on publication day, making the dashboard visible to the public.
-
If you are publishing a new dashboard for the first time that uses unpublished data, then you should have followed the guidance on using dummy data. This means that the unpublished data should not be added to GitHub until the day of publication. You should follow steps 1-9 in the below section on the day before and day of publication.
-
-
Be sure to read the guidance carefully, do not commit or push unpublished data to a GitHub repo before the day of the publication of the data. If you think you may have done this by accident, contact Statistics Deveopment Team immediately with the full details of what has been uploaded to GitHub and when.
-
-
-
-
Updating data in a dashboard
-
-
As mentioned in the public dashboards section, a public dashboard should not be updated with unpublished data until the data is published. However, it is possible to clone the repo and run it locally with unpublished data for testing purposes (see our guidance on testing with unpublished data). This guidance applies to both adding real data to a dashboard that previously used dummy data, and to adding updated data to an existing dashboard.
-
On the day prior to publication:
-
-
Create a local branch which you will use to update the data. To do this open your local version of the repo, navigate to the Git window, ensure the current branch is main (see 1) as this is the code producing the current live version of the app. If you’re happy the current branch is main, then do a pull to ensure it is up-to-date (see 2). Then click the new branch button (see 3).
-
-
-
-
Give your branch a descriptive name, for example, for a data update for publication on a given date, incorporate the date into the branch name (i.e. data_update_01Jan_2023). Click create.
-
You can now add your unpublished data files to the data folder in your local copy of the repository (in the file explorer). If you have already added the data using .gitignore as described in the guidance on testing with unpublished data, you can now remove the file name from the .gitignore file. You can also remove any dummy data files or old data files from the folder.
-
Now open the script that reads in data for your dashboard and edit the file paths to point at the new data files.
-
run tidy_code() and run_tests_locally() (see guidance on UI tests). If changes are found when running the tests locally, make sure you look through these differences and understand them. If you are happy with the changes found in the tests (i.e. they are expected due to your updates), update the .json files. These are tested via GitHub actions every time you do a pull request, so you should always run them in advance of any pull request.
-
You can run your app to test that it is working, but do not commit or push this branch yet.
-
-
On the day of publication:
-
-
At 9:30 on publication day, commit and push the changes in your new branch to GitHub. This makes the data files publicly available via GitHub and so should not be done until 9:30 on the day of publication, however this step does not update the dashboard.
-
On the GitHub repo, you should now see a prompt to open a pull request. You should do this, following the template, ticking the boxes to show you have completed tests locally as required and provide details of the changes (i.e. that you’ve updated to include the most recent data, including dates).
-
As you have already completed tests locally (step 5), you do not have to wait for the tests to complete when you open the pull request, you can click to merge into main straight away. This should start the process of updating the data on the dashboard. You can view the progress and time taken to do this by opening the GitHub actions tab and looking at the shiny-deploy action. Once the shiny-deploy action is complete, the dashboard will have updated.
-
-
-
-
-
-
-
DfE Dashboard template
-
-
Our template shiny app repository should be used a starting point for all public facing dashboards as it gives a consistent set up for publishing and provides template code for common parts. You can start your own repository using the ‘Use this template’ green button on the template repository. You will need a GitHub account to do this. If you are unsure or need help, contact the Statistics Development team who will be able to walk you through this.
-
-
-
What’s in the template
-
The template provides code for a basic interactive dashboard, with an example line chart using drop-downs for education level and geographic levels. It also includes an example accessibility statement (this is a requirement for all public dashboards and must be filled in, see the Accessibility and Accessibility testing sections earlier in this guidance), and a ‘Support and feedback’ page with suggested contact information and links.
The template has a google_analytics.html file which is set up to track all basic metrics, plus the inputs from both drop-downs. To set this up with your own app you will need a new tracking tag (the long number in the gtag()) which the statistics development team can provide you with. Please contact us in order to set this up.
-
If you are hosting your dashboard at multiple URLs (i.e. to cope with expected high traffic) then all URLs will be tracked using one tracking tag and analytics for all URLs will all appear as one in the same report.
-
-
-
-
Data files log
-
-
The datafiles_log.csv file is a record of all of the data files saved in the repo, and whether or not the data is published. You should log all data files here and accurately record whether the data is published, unpublished or reference data.
-
-
-
-
Stopping accidental data uploads
-
-
If you see any of the following errors, it is because you are trying to commit a data file to the GitHub repo that hasn’t been recorded as published:
-
Error: data/test_data.csv is not recorded in datafiles_log.csv.
-
Error: data/test_data.csv is not logged as published or reference data in datafiles_log.csv and is not found in .gitignore.
-
The template uses commit hooks and a datafiles_log.csv file. Commit hooks run automatically whenever you try to commit changes, and prevent any unpublished data being uploaded to the GitHub repo (see the public dashboards and dummy data sections) by checking the datafiles_log.csv file.You should therefore record all data in the datafiles_log and record whether it is published, unpublished or reference dara. If you try to commit a file that is not recorded or is recorded as unpublished, git will not allow the commit. If you would like to save an unpublished data file locally to test the dashboard you should use the .gitignore file to ensure git ignores this file in your commits (see the testing with unpublished data guidance).
-Guidance for how to use the features in the Explore Education Statistics platform
-
-
-
Explore Education Statistics (EES) is the Department’s official statistics dissemination platform, designed to make DFE’s published statistics and data easier to find, access, use and understand.
The admin website, where production teams create their stats
-
-
-
The majority of the admin side of EES is only supported in Google Chrome and Microsoft Edge and will not work correctly in Internet Explorer. The public website and pre-release previews of releases in admin can be viewed using Internet Explorer, though we still recommend that people use Chrome or Edge instead.
-
-
Both applications were released to Minimal Viable Product (MVP) standard in March 2020, we will be iterating and improving the functionality based on user feedback throughout 2020 and 2021.
As stated above, EES consists of two parts. We also have four versions (environments) of EES, the banner for the admin part of each environment will inform you which environment you are on, this is also colour coded:
-
-
Development - Green - where changes are first merged in, and often the first time different pieces of work from different developers will interact properly
-
Test - Pink - where our developers can carry out manual testing of any new features to make sure things work as expected
-
Pre-production - Yellow - a sandbox area for analysts to carry out functionality testing
There is no overlap between the environments and content created on one cannot be moved to any other.
-
-
If you are bookmarking links, please be careful to bookmark the links below exactly as they are shown. Often when signing in you will be redirected via other url’s as a part of the authentication process, and bookmarking those may lead to errors.
-
-
Teams can use the pre-production environment to familiarise themselves with the platform and test out what is possible. The core functionality across the environments is identical except for new changes, which are deployed through the different environments before they make it to production.
-
On the pre-production environment, all analysts have full permissions to create publications and releases, and can see everything else that other analysts are making. This is unique to the pre-production environment, on the production environment analysts will only have access to releases that they have been granted specific access to.
-
When accessing the pre-production environment you may be asked for a username and password, these are as follows: dfe, dataresearch.
-
-
The pre-production environment is not suitable for unpublished data. Unpublished data should only be uploaded to the production environment.
-
-
-
Getting access to admin
-
Access to the production and pre-production admin services is limited to DfE AD accounts only and users have to have been invited to the service by either the Statistics Development Team (for full access) or an existing user (for pre-release access). Invites to the service are sent out via email using gov.uk Notify.
-
To be invited to the service for full access teams need to email the Explore Statistics Mailbox, stating who needs access, what permissions they require (analyst or approver), and for which publications these apply. This email should be sent by the Team Leader, or accompanied with the relevant Team Leader’s approval. Once access has been granted you will receive an email inviting you to use the platform.
-
-
Jobshare emails can not be used to access EES. Specific personal emails should be used instead, in the same way as you sign in to windows on your machine.
-
-
-
-
-
-
Roles and permissions
-
The Statistics Development Team are responsible for setting up and maintaining user permissions during the beta phase. Change requests will be monitored via their main mailbox - explore.statistics@education.gov.uk.
-
The following roles exist within EES admin and are assigned to a specific release:
-
-
Pre-release
-
Any user invited for pre-release is given a pre-release user role. During the hours of pre-release they can:
-
-
Preview the release page, including downloading files
-
Preview the table tool page for that release
-
-
-
-
-
Analyst - contributor
-
Any analyst working on a release within EES admin will have the contributor role. They can:
-
-
Edit release details
-
Upload data and other files
-
Create footnotes, datablocks and charts
-
Edit and comment on release content
-
Invite PRA users
-
Move the release status between draft and ready for higher review
-
-
-
-
-
Analyst - approver
-
The responsible statistician for a statistics release will have the approver role, allowing them to sign off releases for publication. This will usually be the responsible G5 or G6 for the statistics publication. They can do everything a contributor can, as well as:
-
-
Move the release status between draft, ready for review and approved
-
Schedule the release for publication (either immediately or for a specific date)
-
-
-
-
There is also a publication owner role, which is assigned at publication level to a users account:
-
-
Publication owner
-
This permissions level gives publication owners control over their publications. They can do anything a contributor can for all releases within the publication. On top of this the publication owner also has access to:
-
-
Manage publication level details
-
Create new releases within a publication
-
Create an amendment of the latest published release within a publication
-
Cancel an amendment before it is published
-
-
-
-
-
BAU
-
The administrative role:
-
-
This role is assigned to a user and gives full access including administrative tools, typically only used by the Statistics Development Team and EES developers.
-
-
-
See below for a diagram of the responsibilities of each role as part of the publication process in EES:
-
-
-
-
-
Requesting a new publication
-
If you have not published on EES before, or if you’re creating a brand new publication, you will need to contact us to create the publication for you before you can get started.
-
The hierarchy of content within EES is as follows -
-
-
Publications are organised into topics and themes (as shown on the EES Find Statistics page), and then within each publication there are releases - where the latest release includes the latest statistics for that publication. For example:
-
-
-
-
Level
-
Example
-
-
-
-
-
Theme
-
Pupils and Schools
-
-
-
Topic
-
Pupil absence
-
-
-
Publication
-
Pupil absence in schools in England
-
-
-
Release
-
2018/19
-
-
-
-
Each publication should have a lead statistician’s name and contact details attached to it as a requirement of the Code of Practice. You may use a team email address but the name and telephone number of a named statistician should also be provided.
-
Methodology documentation is also attached at a publication level within EES - meaning one standalone piece should be written to cover all releases for the given publication within the service.
-
If you want to request a new publication, please be prepared to give the following:
-
-
The heading and subheading that you want the publication to sit under in our site (take a look at the current structure here), feel free to either use existing ones, or suggest new ones
-
The exact title of the publication
-
A summary of less than 160 characters (including spaces) that describes your publication series
-
If the methodology already exists elsewhere and you have a URL for it, or if you want to create one
Use Google Chrome or Microsoft Edge to access and use the admin part of Explore Education Statistics.
-
-
When you enter the service you’ll see the admin dashboard, here what you can see will be dependent on your access permissions i.e. you’ll only see the publications that you have been granted access to. Within the test environment users are set up with full permissions (i.e. they can see/do everything) though in our production environment this will be restricted.
-
The Statistics Development Team will be responsible for setting up and maintaining user permissions during the beta phase. Change requests will be monitored via their EES mailbox - explore.statistics@education.gov.uk.
-
-
Within this dashboard you can view and manage existing publications, including creating and editing their releases. You can use the drop down lists to find releases by theme/topic/publication or use the draft and scheduled releases tabs to see releases that are in progress.
-
-
-
-
External user access
-
Access to EES for users outside of DfE is difficult as the authentication and security on the service relies on the DfE Azure Active Directory and associated infrastructure. It is possible to request that external users from other government organisations have access, and they can then be added as collaborating analysts on a release or as pre-release viewers.
-
If you have external users you’d like to request access for, please send the following to explore.statistics@education.gov.uk, at least two weeks in advance of requiring access. Due to the dependency on DfE’s digital security we cannot guarantee access or how long it may take.
Once you have created your release you will see enter the ’create release dashboard, here you need to work through the following tabs to create your release:
-
-
Data and files (uploading data files, ancillary files and creating data guidance)
-
Footnotes (creating footnotes and assigning to relevant data)
-
Data blocks (creating summary tables, charts and key stats indicators)
-
Content (drafting release content)
-
Sign off (moving through the approval process)
-
Pre-release access (invite users to pre-release and create public pre-release list)
-
-
-
-
-
Data and files
-
Here you will update your data files and accompanying metadata. You need to make sure that the data has passed through our data screener checks before trying to upload it.
-
-
All data files are accessible in the downloadable files for users to explore in the same format as they are uploaded.
-
-
-
-
Subject titles
-
-
You’ll need to give a ‘subject title’ to each data file you upload. The subject name should be a simple user-friendly title for the data included within the file - it is what users will see wherever the file is referenced within EES. The actual file name and data guidance can include more technical / coverage information.
-
When adding a subject name think about the general user and how the subjects will appear in the service:
-
-
You don’t need to include the publication name in the subject name as this is always already implied within EES
-
You don’t need to list what filters are in each file in the title, users can see this in the data guidance
-
You don’t need to include the date ranges covered in each file in the title, users can see this in the data guidance
-
Your subject names should be short and snappy, clearly explain what is in each file. Some good examples of this in practice are included below:
-
-
Early years provision by provider type
-
Exclusions and suspensions by pupil characteristics
-
ITT new entrants by subject and training route
-
-
-
You can make changes the subject title for your data file after it has been uploaded if needed using the ‘Edit title’ option.
-
-
-
-
Uploading files
-
-
When uploading files you have a choice between uploading as separate CSV files or as a combined ZIP file.
-
For data files greater than 80mb we recommend uploading as a ZIP file.
-
Once you click to upload the file a ‘Status’ will be visible that shows the progress of the import process. It will go through the following stages:
-
-
Queued
-
Validating
-
Importing
-
Complete
-
-
This may take a little while depending on the size of your file and if there are numerous files queued for import. You cannot view the dataset or use it to create tables/charts until this status is ‘COMPLETE’.
-
-
If you believe your file is ‘stuck’ please contact explore.statistics@education.gov.uk, with details of the file and the release that you are uploading to.
-
-
-
-
-
Ordering filters and indicators
-
-
To save time when creating tables and charts, and to aid users who view your data themselves via the table tool, you can now save custom orders for your filters and indicators. You can do this in the data and files page after uploading underlying data files, via the ‘reorder filters and indicators’ tab. Once there just choose which file you want to reorder and then click and drag the items until they’re in the order you want them to show in the table tool. This then becomes the default order for this filter or indicator and will apply in all charts and tables automatically.
-
-
-
-
-
Ancillary file uploads
-
-
Any files you want to make available for users to download but aren’t intended the table tool should be added as an ancillary file upload. These files will need to meet all requirements of the new accesibility regulations before they can be published.
-
Examples of ancillary files may be:
-
-
Infographic pages
-
Supplementary data that isn’t intended for the table tool
-
Additional technical documentation
-
-
For ensuring that spreadsheets are accessible see the guidance from .gov.wales, and this blog post that walks you through some ways that you can recreate the experience of using spreadsheets.
-
-
Any data files that only consist of Planning area, and/or Institution level data should be uploaded as an ancillary file, rather than as a data file.
-
-
-
-
-
Public data guidance
-
-
Here you should also create your public data guidance document, this replaces the information that would have previously been uploaded as a pdf and is designed to help users understand and use the data they download from your release.
-
-
The document will automatically update as you add new data files to your release, however you will need to add an overview of all the data included in the release and short summaries for each data file before the release can be published.
-
A list of variables in each file with an associated label (taken from metadata uploads) and associated footnotes will also be displayed for each file.
-
-
-
Releases cannot be published without a completed metadata document. If not filled in an error will be flagged during sign off.
-
-
-
-
-
Replacing data
-
-
If you just need to change the subject title for your data file you do not need to go through the whole replacement process, just click the ‘Edit title’ option.
-
However, if you notice a mistake in your data file you can replace it with another. When replacing a data file the platform will attempt to recreate any data blocks and footnotes that were built using the previous file.
-
-
The replacement file must contain the exact same column names and types as the original. For example, a character column named “date” must also be replaced with a character column named “date”. A numeric column named “date” will not work in the replacement.
-
-
-
The first step is to upload the new file.
-
-
One you’ve chosen and uploaded your replacement file it will need to go through the usual import process before it can check if retaining existing data blocks and footnotes will be possible.
-
-
Once the upload is finished a report will appear which highlights whether existing data blocks and footnotes can be replaced successfully. If you want to keep any data blocks and footnotes you’ve built you will need to make sure that your replacement data file still contains the information (indicators, filters, geographic_levels and time_periods) that was used to create them.
-
-
If it’s not possible for a data block or footnote to be recreated using the replacing data file a warning will appear and you’ll be prompted to either edit or delete them before completing the replacement.
-
-
-
Remember to double check any data blocks or footnotes that were recreated by the platform before publishing your release.
-
-
-
-
-
-
Footnotes
-
We generally advise against using footnotes, unless they’re absolutely necessary (think about how many people will actually be reading them!). Consider where you could signpost to the methodology /data guidance / commentary instead. However, if you do want to add footnotes to your data, this should be done using the EES footnote function.
-
Footnotes are added via footnotes tab - rather than writing multiple tables and assigning individual footnotes, you write footnotes and assign them to certain indicators and filters so they appear when users select them in the table builder. For example in the below, the footnote “This is a footnote” is assigned to “Headcount” indicator for all options within the “School type” filter.
-
-
If you’d rather, you can assign a footnote to the whole data file by ticking this box.
-
-
You can assign the same footnote across multiple data files.
-
-
We recommend that you only add footnotes once you are certain the data file is final. If you have to delete the data file, all the assigned footnotes will be deleted alongside it.
-
-
-
-
-
Data blocks (tables and charts)
-
A data block is a smaller cut of data from your original file that you can embed into your publication as a presentation table, build charts from, or link users directly to.
-
-
-
Tables
-
-
Here you can create data blocks, using the admin table tool to create and save your summary tables. Remember to use the reorder table headers tool to restructure your table however you want it before you save.
-
Once you’re happy with your table, give it a title and some source information before clicking save. The source should be the source of the data used to create the data file that was then used for the data block table, for example ‘School Census’.
-
The manage data blocks page will list all the data blocks you have created, highlighting which have charts, are used in content and are saved as highlight tables.
-
-
-
-
Featured tables
-
-
You can also choose to highlight a data block table as a ‘featured table’ which means it will show in a list of featured tables within the table tool. This is designed to help users get to tables of interest more quickly (without having to create tables themselves).
-
There is an option to choose if a table is a ‘featured table’ when saving each data block, here you can name the table and add a description giving the table coverage:
-
-
Each featured table will then be listed to the user within the table tool. Featured tables do not have to be embedded within your release content to be included in this list.
-
-
-
-
-
Fast track links
-
-
Any data block tables that are created and saved within the admin app will also be assigned a ‘fast-track link’, this url can then be used throughout your release as a way to direct users to specific tables (within the table tool) more quickly so they can interact and explore the data further. It will appear at the top of the page like this:
-
-
When your release is published, any embedded data block tables within the release will have an ‘explore data button’ beneath them which will utilise these fast track links to quickly direct users to the table within the table tool so they can explore the data further. You can also use fast-track links as a hyperlink within release commentary (without having to embed the data block).
-
In your fast-track titles, you don’t want to overload information, but still want to direct the user to the right place. Remember they can go back to the table tool through your featured tables to change filters and indicators as needed.
-
Fast-track titles should explain:
-
-
What the table is showing in the simplest terms
-
Who/Where the data covers (e.g. characteristic groups and geography levels)
-
When the data in the table is reported for
-
-
Here are a few examples of good fast-track titles in EES:
-
-
Number of Schools and Pupils, by School Type, 2015/16 to 2020/21
-
Absence Rates by School Type, 2016/17 to 2020/21
-
Free School Meals, by Region, 2015/16 to 2020/21
-
-
-
What is the difference between a fast-track link and a permalink?
-
EES also offers ‘permalinks’ for any table created in the table tool which allows a user to save a link to a permanent, static, version of a table they have created. Analysts can make use of these permalinks when answering queries or in PQ and FOI responses.
-
Fast-track links are similar to permalinks however instead of linking to a static version of a table they link to an ‘active’ version of the table within the table tool - meaning users can interact and change what’s shown in the table from within the table tool if they choose to.
-
-
-
-
-
-
Charts
-
-
After building and saving a data block table you will see a ‘Chart’ tab appears. This tab will take you to the EES chart builder, where you can choose to add a chart to your data block.
-
The first step to creating a chart is choosing the chart type, currently the EES chart builder can build line charts, horizontal/vertical bar charts (including stacked and clustered) and maps.
-
After choosing your chart type you then need to work through the following stages to build your chart. In each stage you’ll be shown a live preview as you make changes.
-
-
-
Chart configuration
-
-
Within the ‘chart configuration’ tab you can add a title, alt text, move the legend and change chart dimensions.
-
-
Make sure to review your chart dimensions before you publish. Users should be able to read the labels on the axes and see the legend without having to scroll.
-
-
-
Note, within the vertical and horizontal bar chart types you can also create stacked bar charts by clicking the ‘Stacked bars’ option within the chart configuration tab
-
-
-
-
Data sets
-
-
Here is where you add data to the chart. You can add each series one at a time or all together.
-
-
-
-
-
Legend
-
-
You can edit the chart legend, and styling of your series via the Legend tab.
-
-
-
To select specify custom colours outside of the defaults, you can double click on the colour codes at the bottom of the colour picked until you get to the type of code you’re wanting to input (e.g. hex code) and then enter the code manually.
-
-
-
-
-
X axis (major axis)
-
-
Here is where you configure the x-axis: You can alter gridlines, labels, sort, limit and add reference lines.
-
-
-
-
-
-
-
-
-
Option
-
What it does
-
-
-
-
-
Size of axis
-
Change the width of the space given to axis tick labels
-
-
-
Show grid lines
-
Turn grid lines on and off
-
-
-
Show axis
-
Turn the axis on and off, you can also add a unit to the axis tick labels
-
-
-
Sorting
-
Change how the data within the chart is sorted
-
-
-
Tick display type
-
Alter how often axis tick labels are shown, labels will automatically skip values where there are too many to show without overlapping
-
-
-
Axis range
-
Alter the range of data shown in the chart
-
-
-
Label
-
Add an axis label, you can also choose the width for the space given to it
-
-
-
Reference lines
-
Add/remove reference lines to the chart
-
-
-
-
-
-
-
Y axis (minor axis)
-
-
Then follow a similar process for the y axis configuration, play around until the chart looks how you want it to.
-
-
-
-
-
-
-
-
-
Option
-
What it does
-
-
-
-
-
Size of axis
-
Change the width of the space given to axis tick labels
-
-
-
Group data by
-
Change how the data within the chart is grouped
-
-
-
Show grid lines
-
Turn grid lines on and off
-
-
-
Show axis
-
Turn the axis on and off, you can also add a unit to the axis tick labels
-
-
-
Sorting
-
Change how the data within the chart is sorted
-
-
-
Tick display type
-
Alter how often axis tick labels are shown, labels will automatically skip values where there are too many to show without overlapping
-
-
-
Axis range
-
Alter the range of data shown in the chart
-
-
-
Label
-
Add an axis label, you can also choose the width for the space given to it
-
-
-
Reference lines
-
Add/remove reference lines to the chart
-
-
-
-
-
-
-
Changing chart type
-
-
If you create your chart and then change your mind as to what chart type would be best you can just click to change it and it will try to save all the options that you had applied previously.
-
-
-
Remember to save your chart when you’re done.
-
-
-
-
-
Maps
-
-
You can create maps too, currently this is possible for regional, LA and LAD data.
-
You can change the boundaries you are plotting onto via the “chart configuration” tab, the latest boundary file will automatically be selected, but if you are visualising historic data, you may want an older boundary file. Please contact us if the boundary you want to plot is unavailable.
-
-
Then to create a map, add the cuts of data you want to display in the “data sets” tab of the chart builder.
-
-
You can change the colour scale of the chart in the “legend” tab.
-
-
-
-
-
Other chart types
-
-
You should only use complex charts where there is a clear user need, as simple charts are the easiest for users to understand. If you have a an infographic or a chart that is too complex to build within our chart builder you can use the ‘Choose an infographic as alternative’ option to upload an image to your data block or use the Add embed block feature to embed an R-Shiny based plot (see the section on embedded blocks for further information).
-
Infographic alternatives should be .svg format and you can use the sizing options within the data block editor to size your image appropriately.
-
-
Accessibility matters for infographics too, consider the following if you do need to upload an image -
Try to avoid adding text to your images other than axis labels and limited annotations. Titles and headings can be added in the “chart configuration” tab instead
-
-
-
You can use R to create infographics and the following code gives an example of how to create a basic line chart or stacked bar chart using the appropriate GSS sequential colour palette.
-
# Load the necessary libraries
-library(ggplot2)
-library(dplyr)
-
-# This is the GSS standard categorical colour palette
-gss_categorical_palette <- data.frame(
- names = c('Dark Blue', 'Turquoise', 'Dark pink', 'Orange', 'Dark grey', 'Light purple'),
- hex = c("#12436D", '#28A197', "#801650", "#F46A25", "#3D3D3D", "#A285D1"),
- id = c(1,2,3,4,5,6))
-
-# Set up a dummy data-set.
-# Note the line factor(...,levels=...) allows you to order your filter values in
-# the final plot based on the ordering entered into the levels keyword. If left,
-# it'll default to alphabetical.
-data <- data.frame(
- time_period=c("2018/19","2018/19","2018/19","2018/19","2018/19","2018/19",
- "2019/20","2019/20","2019/20","2019/20","2019/20","2019/20",
- "2020/21","2020/21","2020/21","2020/21","2020/21","2020/21"),
- Filter=c(gss_categorical_palette$names,
- gss_categorical_palette$names,
- gss_categorical_palette$names),
- indicator1=sample(4:16,18,replace=TRUE) +
- c(gss_categorical_palette$id,gss_categorical_palette$id,gss_categorical_palette$id)*3
- ) %>%
- mutate(
- time_period=as.factor(time_period),
- Filter=factor(Filter,levels=gss_categorical_palette$names)
- )
-
-# Create a line chart
-ggplot(data, aes(x=time_period, y=indicator1, group=Filter, colour=Filter )) +
- geom_line(size=1.2) +
- scale_color_manual(values=gss_categorical_palette$hex) +
- theme_classic() +
- theme(
- legend.position = "bottom",
- text = element_text(size = 14, family = "Arial"),
- strip.text.x = element_text(size = 20)
- ) +
- ylab("Indicator 1") +
- xlab("Time period")
-
-# Create a stacked bar chart
-ggplot(data, aes(x=time_period, y=indicator1, fill=Filter )) +
- geom_bar(stat = "identity") +
- scale_fill_manual(values=gss_categorical_palette$hex) +
- theme_classic() +
- theme(
- legend.position = "bottom",
- text = element_text(size = 14, family = "Arial"),
- strip.text.x = element_text(size = 20)
- ) +
- ylab("Indicator 1") +
- xlab("Time period")
-
The above code should produce something along the lines of the following plots:
-
-
-
-
-
-
-
Content
-
In the content tab you can now start creating your release, embedding the data blocks you’ve created as you go.
-
-
You can use the page view toggles, that float in the bottom left of the page, to jump between edit and preview mode for the release and to view a preview of the table tool.
-
-
Add any headline and/or key stats and figures for your release in the headline facts and figures section.
-
You can then create accordion sections to start adding your main release commentary. These sections are made up of text blocks and data blocks which can be reordered as needed.
-
-
Headline facts and figures
-
Use the ‘Add secondary stats’ button to add a data block to your headline stats section.
-
Use the ‘Key stats’ options to add key statistic tiles to your release. For each tile you first have to have created a data block that contains only one number.
-
-
After embedding a key stat tile you can then edit it to add trend information and a description of what the indicator is.
-
-
-
-
-
-
-
-
Element
-
Content
-
-
-
-
-
Indicator name
-
Automatically generated from your data
-
-
-
Latest value
-
Automatically generated from your data
-
-
-
Trend
-
A short one-sentence description of the trend; try to avoid only stating the change from the previous year and talk about the longer-term trend where appropriate
-
-
-
Guidance title
-
E.g. ‘What is NEET?’ or ‘What are permanent exclusions?’
-
-
-
Guidance text
-
A simple description in plain English of what the indicator is
-
-
-
-
-
-
-
Drafting text
-
-
-
About these statistics
-
-
We recommend that the first accordion section in each release should be ‘About these statistics’. Do not assume that users will read it, the nature of interactive pages means that the reader may start at any accordion section so remember to include essential information in the section to which it refers.
-
Avoid filling this section with:
-
-
Crucial caveats or information necessary for accurate interpretation of the statistics, these should be included within the main commentary next to the point they relate to
-
Technical information that is not relevant to the interpretation of the statistics should be saved for the methodology
-
Definitions should be included within the main commentary when they are first mentioned
-
Lists or descriptions of what is in the commentary as these should be clear from the contents and accordion headings
-
-
Try to focus the ‘About these statistics’ section on:
-
-
Say why the data are collected and what they are used for or could be used for, including relevant policies and targets
-
The different geographical levels for which data is published
-
Non-essential but important information about the statistics, such as:
-
-
The data collection window, mode of data collection and response rates in terms of whether these are representative
-
Clarification of the population coverage
-
Changes in data collection or analysis methods from previous releases
-
-
Links to relevant related statistics including the cause of any differences
-
Further useful detail on specific measures used in the release
-
Non-urgent notices to users such as minor error corrections and delays or changes to the next publication. If there is something important which you want all users to know about then give it its own accordion section.
-
-
-
-
-
-
Footnotes
-
-
-
Do not use footnotes in the text of your content. They’re designed for reference in print, not web pages. If the information in the footnotes is important, include it in the body text. If it’s not, leave it out.
-
-
-
-
Tables in content
-
-
Any data tables should be included as data blocks, however you can also embed static html tables within a text box. These should only be used to present textual tables or for any small presentations of data that are not possible to do in a data block at the moment.
-
-
-
You can create static html tables for presenting information that isn’t embedded in a data block. However, remember that all of the data included or referred to in your content should be available (or createable) from the downloadable open data files.
-
-
-
-
Footnotes for tables in content
-
-
If you are including a table in text that needs footnotes, it’s generally advised to include this in the commentary surrounding the table. However, if you think a footnote is still necessary, then we advise writing out the word ‘note’, with the number of the note you need to refer to, and put it in square brackets, for example: ‘Number of people in employment [note 1]’. For more guidance on footnotes outside of EES, see the Analysis Function guidance on symbols, footnotes and codes.
-
-
-
-
-
Reviewing content
-
-
While a release is in draft mode, comments can be added to text to help teams collaborate. Simply highlight the text you want to comment on and click on the speech bubble in the editing bar to add a comment.
-
-
Comments can be edited at a later date, and can also be marked as resolved so that you can see which comments have been addressed and which are still outstanding.
-
When someone is editing a text box, it will now be instantly frozen for all other users preventing two users from editing the same block of text at the same time. You will be able to see the name of the user who is making edits, and will see the edits coming through every few seconds as their changes autosave.
-
-
-
-
-
-
Embedding R-Shiny blocks
-
If you need to include a type of chart that isn’t possible using the standard EES chart options, for example an interactive chart with filters, you can embed a block to display a custom R-Shiny produced chart.
-
To embed a shiny chart, you’ll first need to create an R-Shiny application containing the chart using the DfE tiny-shiny template and get it hosted on our DfE GitHub and ShinyApps accounts. More details on both of these are available in the tiny-shiny section of this guidance.
-
Once you’ve got the R-Shiny app set up and hosted, you can embed it using the Add embed block button in the content area. This will give you the option to enter a URL, where you can enter the URL of the shiny app. Valid URLs to use in the embed block dialogue box are limited to only those on the DfE ShinyApps server (and the internal rsconnect when needed for draft publications).
-
-
-
-The EES interface for embedding a Shiny chart
-
-
-
-
-
-
Glossary
-
Our glossary on EES is a growing page that helps us to standardise how refer to key terms and phrases across all of Official statistics - https://explore-education-statistics.service.gov.uk/glossary.
-
-
-
Adding new definitions
-
-
Contact us at explore.statistics@education.gov.uk, with the title and definition and we can add this for you. It’s worth having a check on the glossary for similar or related terms, and whether you should be looking to harmonise with other teams.
-
-
-
-
Linking to definitions
-
-
You can link to any term in the glossary by appending the glossary url with #name-of-definition, replacing any spaces with hyphens.
-
For example, to link to the definition for ‘Respite care’, you would use the following link:
You can test this works by typing the url into your browser, it should take you to that specific definition on the glossary page. If you’re unsure at all, or have special characters in the title of your glossary entry, please ask us for support on getting the right link.
-
-
-
-
Linking from your release
-
-
When writing your release content, highlight a word or phrase and click the link icon in the text editor bar.
-
-
Then, paste your glossary url into the box that appears.
-
-
That’s it, the system will automatically recognise that the link is for the glossary and will do the rest. You can then change to the preview mode to see how this would then appear to public users, and test that the box appears with the definition.
-
-
-
-
-
-
-
Sign off
-
Once you’re happy with your release you need to go to the sign off page and change it’s status in order to move it through the release pipeline.
-
There are three statuses:
-
-
In draft (where the production team work on drafting the release)
-
Ready for higher review (where the senior statistician checks over the release before approving)
-
Approved for publication (after approval has been given, releases in this status will be published on their scheduled date)
-
-
-
Only users with approver permissions (usually G6 or above) can sign off the release for publication.
-
-
-
-
Errors and warnings
-
-
There’s quite a few things to remember to do as you build your release so to help ensure you haven’t missed anything a release checklist is also available via the sign off page
-
The checklist can be accessed by clicking to edit release status.
-
Remember to check over it before you submit your release for approval as a release that has outstanding warnings on it will not be able to be published.
-
-
-
-
-
How to approve a release
-
-
Releases are approved via the sign off tab. The release date is also set during this stage of the process. All releases scheduled for a specific date will be published at 9:30am on that day.
-
-
The approver has the ability to approve the release to be published on a specific date or to publish as soon as possible. Publishing as soon as possible is useful for publishing amendments to existing releases.
-
-
This page also gives you the expected release url which may be useful to know for other things, for example, sending to the web team to add to your gov.uk announcement page.
-
-
-
-
-
Next release expected
-
-
You also have the option to add a date for when the next release is expected. This will appear at the top of a release page and give users an idea of when to expect the next release in the publication series. You can provide the planned month and year of the next release within this publication.
-
If this is added but then needs to be changed or removed later in the year it is easy to do so, by creating an amendment and setting a new next release expected date on sign-off.
-
-
-
-
-
Pre-release access
-
Prior to each release going live the production team are also able to grant pre-release access to a named group of users 24 hours before it goes live. These users do not require full access to the whole admin service. They will be able to see preview versions of any releases they have been granted access to.
-
This preview is only accessible for the 24 hours before the publication date, the emails may go out to users before then. We expect teams will continue to send an email at 09:30 on pre-release day, including any additional briefing and the link to the pre-release area.
-
-
Jobshare emails should not be sent invites for pre-release. The personal emails for the specific individuals should be used instead, as jobshare emails do not have active accounts with DfE to access EES.
-
-
-
Pre-release access via Explore Education Statistics is currently limited to DFE users only, if you need to share your release with external users you will need to do so outside of the system.
-
-
-
-
-
-EES PRA one pager
-
-
-
-
-
-
-
-
-
-
-
-
Inviting users for pre-release access and building the public pre-release list can be found within the ‘Pre-release access’ tab on the dashboard.
-
Once the release has been marked as approved, go to the ‘Pre-release access’ tab and add the relevant email addresses to grant pre-release access. All invited users will receive an email to say that they have been given pre-release access and will get a url where the preview release will be available.
-
-
You should also create the public facing pre-release access page by clicking the ‘Public access list’ tab.
-
-
After creating your pre-release access list a text editor will appear where you can list the roles that have been given early access to the release. This list will then appear in the list of file downloads at the top of each release.
-
-
-
-
-
-
Release amendments
-
-
If you need to make an amendment to release that has already been published this is possible to do from within the admin dashboard if you are a publication owner. If you need to create an amendment and your publication owner is not available, please contact the Statistics Development Team to do this for you.
-
-
Creating an amendment of a release will create a new version for which you can make any changes and then choose to republish, either as soon as possible or on a given date.
-
-
-
Release notes
-
When publishing a new amendment you should add a ‘release note’ to your release so users can be informed of what has changed:
-
-
-
-
-
-
Methodology
-
-
Methodologies work in a very similar way to the written content of a release, the text editor and static tables you can use are built on the same foundations.
-
If you have not created or published a methodology page on EES before, the first thing your publication owner will need to do is create a methodology page.
-
-
Once created you can then edit the methodology content in a similar way to how you would create release page content.
-
Methodologies should follow our recommended methodology template, to ensure all the essential details are covered.
-
Once the drafting has finished, a methodology should be set as ‘approved for publication’, currently approved methodology changes will be published immediately so make sure you are ready for the changes to go live before approving.
-
If you want to revise an existing methodology page you can amend the methodology which will generate a new version for you to edit.
-
-
Once you’re happy with your amendment it just needs to be approved in the usual way for it to be published.
-
-
-
Methodology formatting
-
Subheadings make it easier for your users to navigate through your methodology. When editing text blocks in your methodology, you can do this by highlighting your subheading and selecting “Heading 3” in place of “Paragraph”:
-
-
You can add further subheadings underneath this but try to limit the number of subheadings per accordion to 10 at the most. Too many subheadings will make it tricky for users to find what they are looking for.
-
-
-
-
Methodology images
-
You can add images to methodology pages via the content editor.
-
-
When uploading an image you will need to add alt text via by clicking the eye symbol, you may also choose to add an image caption.
-
-
-
Accessibility matters for uploaded images too, consider the following if you do need to upload an image -
Ensure there is a high enough colour contrast ratio between elements. You can use a colour contrast checker to check the colours you’re using
-
Try to avoid adding text to your images other than axis labels and limited annotations.
-
-
-
-
-
-
-
Managing publications
-
-
We have a type of role within EES admin called ‘Publication Owner’. This permissions level gives publication owners control over their publications.
-
Options available to publication owners:
-
-
-
-
Publication management
-
Use this page to change the details for your publication, such as the contact information or to add links to legacy releases on gov.uk.
-
Any updates here will affect all releases in the series and will be published immediately.
-
-
It is possible to change the publication title via this page, however this should be used rarely and will not affect the publications url if one or more published releases exist.
-
-
-
-
-
-
Create new release
-
After finding the publication you want to create a release for, just press the button to create a new release.
-
Currently the following types of release can be created in EES:
-
-
National Statistics (includes OSR tick mark logo at the top of the release page)
-
Official Statistics
-
Adhoc Statistics
-
-
When creating a release you will be asked to fill in some release summary fields.
-
-
The time period for the release should reflect the time period of the data that this latest release adds to the time series.
-
-
-
-
-
Managing legacy releases
-
Past publications on other services can be added to the previous releases links that appear on the top right of a release page by using the ‘Manage legacy releases’ section. In here you can add links to previous releases and choose the order in which they appear.
-
This section can be found by publication owners in the publication management section.
-Guidance for creating and embedding R-Shiny visualisations in EES publications
-
-
-
-
Introduction
-
-
EES now offers the option to add an embedded block, which allows analysts to present custom R created charts within their publications. This necessitates analysts creating an R-Shiny based app on GitHub containing the required chart that can then be hosted on a shiny server (currently using the DfE’s account on shinyapps.io).
-
Note that this current combination of GitHub and ShinyApps for publishing Shiny Apps does not offer the appropriate security required for unpublished DfE data. Unpublished uderlying app data should therefore never be uploaded to GitHub / ShinyApps.io until such time as it has been published (i.e. via EES).
-
Prior to publication, either dummy data or the already published data should be substituted in for the purposes of uploading to GitHub and deploying to ShinyApps.
-
If you need a live version of the dashboard with the unpublished data for pre-release reviews and access, the following options are available:
-
-
Demo the chart in R-Studio
-
Create a copy of the chart repository on Dev Ops, deploy this to rsconnect and use the rsconnect link as the embedded URL prior to publication (note this will need updating to the public ShinyApps link before the publication goes live).
-
-
We are currently putting in place a case to provide an in ternal shiny server platform, which will allow greater control of security around our data and allow draft Shiny applications to use unpublished data for internal use.
-
-
-
When to use an embedded chart
-
EES provides a wide range of inbuilt chart options and will always be the first preference for static line, bar and geographical charts. This helps us to clearly maintain consistent styling and accessibility levels across the site.
-
However, there are some instances where you might want to publish something that we can’t provide through EES. Example use cases would be:
-
-
interactive charts controlled by drop-down filters;
-
chart types not provided by EES, e.g. sankey diagrams, box plots, waffle charts, pie charts, dumbbell charts, etc.
-
-
-
-
-
Tools
-
We currently only support custom charts created using R-Shiny. These should be created with ggplot and plotly. We provide a template example of a demo R-Shiny/ggplot chart on the DfE Analytical Services GitHub site, which is described below.
-
-
-
-
Review and authorisation
-
To get a custom chart approved for embedding within a publication, you’ll need to get it reviewed by the Statistics Development team (in addition to your standard approval chain).
-
-
-
-
-
Creating an embeddable R-Shiny App
-
-
An embedded chart should take the form of a single chart, with necessary input options and nothing else. Avoid incorporating:
-
-
multiple different charts
-
using navigation elements
-
images and logos
-
-
And definitely don’t include unpublished data in any app uploaded to GitHub/ShinyApps prior to publication.
-
Other elements such as tables may also be possible if there’s a clear use case for them that EES alone can’t meet. Get in touch with us if you want to check that what you’d like to include is compatible with the intended use of the embed block.
The template provides code for some basic interactive plots. Each example plot is contained with one of the existing branches below for demonstration purposes:
-
-
demo-interactive-bar
-
demo-interactive-line
-
-
-
-
-
Working with data
-
As with the full dashboards, the embedded charts currently require the underlying data to be either included within the app repository on GitHub or uploaded elsewhere publicly accessible such as Google Drive or Dropbox. This currently means that any embedded charts being developed will need to use either dummy data or previously published data until the moment of publication. At the point of the parent release going live, the chart can then be updated with the latest data. Do not upload unpublished data to GitHub, Google Drive or Dropbox.
-
As described earlier, where you need to use unpublished data in your chart prior to publication, you can either a) run the chart locally in R-Studio (without pushing the unpublished data to GitHub) or b) create a DevOps/rsconnect deploy of your app, which can be temporarily used as the embed block URL. Note, this will need updating to a URL to the public dashboard on ShinyApps ready for publication.
-
We are currently developing a route to allow charts via R-Shiny apps to be hosted on DfE servers, such that draft publications will be able to incorporate embedded charts with the unpublished data. The data itself will then be accessed either from a SQL database on DfE servers.
-
-
-
-
-
Development requirements
-
-
Many of the same principles that apply to creating a full R-Shiny dashboard apply to creating a custom embeddable charts. These are:
Specific design recommendations for embedded plots
-
Example code for producing an embeddable shiny chart is given in the template tiny shiny app repository. The following recommendations should be followed in adapting this code:
-
-
Figures should be produced using plotly/ggplot2
-
Figure lengths and heights should be in the range 6-10cm
-
Text sizes in plots should be no smaller than 12pt
-Examples of good practice in EES, from file names to content
-
-
-
-
Files
-
Files can be downloaded from multiple areas across the platform and often users will judge their contents based on the name alone. For display names, it’s important we make it as clear as possible to users as to what each file contains without cluttering the title with information that is already available from the page around it, saving detailed coverage for the data guidance.
The Education and training statistics for the UK release has some good examples of file names which clearly explain what is in the file, while falling well under 35-50 characters in length, e.g.:
-
-
uk_schools.csv
-
uk_expenditure.csv
-
uk_pupils.csv
-
-
The file names do not have time periods in them, making them easier for users to make use of newer versions in future
The Initial Teacher Training performance profiles publication is one example of needing to upload a file outside of the regular data/metadata structure. Their provider-level tables also contain national and regional data, which is not supported in EES at the time of writing. The file has a clear title, and data guidance clearly explains what is in the file.
-
-
-
-
-
-
Data guidance
-
The public data guidance is a key element of your release, to help direct users to the right file to download or build tables with. It is frequently highlighted by users of our stats as a really helpful element of EES. You will be prompted to fill in data guidance in the release checklist. Any release that does not have a complete public data guidance will not be blocked from approval and publishing.
The School Workforce Census, along with having sensible brief file names, also has great examples of useful data guidance, which explain what is in each file without overloading the user with information:
The NEET annual brief is a good example of a public metadata overview section done well. It follows the EES template, goes into just the right amount of detail, and is easy to understand.
Charts and tables are key ways to highlight important information and stories in your release. They should be clear, contain few footnotes that are essential to interpreting the chart or table, and have good descriptive alternative text so users with screen readers can understand what is being visualised.
The Further Education: outcome based success measures release contains infographics that cannot be built in EES. They are saved as SVGs so render clearly on any device, and follows the same colour palette as the rest of the release.
The Widening participation in higher education publication has a good example of a chart built with a long timeseries in the chart builder, with sparing use of footnotes. For example below, a line in the chart indicates a point where comparisons cannot be made, and a key caveat around the comparability of years is included as a footnote, which is needed to interpret the data and chart accurately.
-
-
-
-
-
Accessibility and alt-text
-
-
Good alt-text descriptions in charts will not just repeat the title of the chart, but instead describe the type of chart, what the data coverage is, and what trends can be seen in the chart to a user accessing the page with a screenreader.
-
The school workforce census has a good example of alt-text for one of their charts. The associated alt-text for the chart below is: “Line chart showing the percentage of all teachers taking absence across all state funded schools in England between the academic years 2014/15 and 2018/19. The chart shows a fall from 55.7% to 54.0% in this period.”
Good featured tables have short, easy to understand titles, and point users to commonly requested tables or other tables that might be interesting to a wide range of users. For example, the pupil absence in schools: autumn term release has a good example of a clear title, with further information (including date and geographic coverage) left in the description. Leaving these out of the main title makes it much easier for users to navigate through multiple featured tables:
-
-
-
-
-
-
Content
-
Clear, concise content is required to direct your users to the right place, and keep them engaged with your release. Our content guidance page contains a raft of great advice on how to structure your release and write for the general public.
The Education, health and care plans release has a good example of a headlines section. There are a sensible number of key statistics, and the summary below gives an overview of trends without diving into the numbers and overwhelming the user.
-
-
The Exclusions release has a good example of custom explanations under key stat tiles to show users exactly what the numbers are describing:
The parental responsibility measures release has good examples of writing for the public in accordion content. Content in each accordion is short and follows the “pyramid” principle of having essential information at the top, summarising the trend, then going into detail at the bottom.
The EHCP release has some great examples of active subheadings within accordions, which explain the overall trends without overloading users with detail.
Good methodologies will broadly cover topics in our recommended methodology template. The Graduate outcomes (LEO) methodology broadly follows this outline, making use of effective formatting in EES to help users navigate through accordions, although definitions can be moved to the EES glossary.
-
The Key stage 4 destinations methodology makes good use of annexes to show changes to their methodology over time.
-
-
-
-
-
Approvals and amendments
-
Keeping track of processes for approving and amending releases is crucial for transparency. Internally, there should be clear paper trail of who has signed off the release, and externally, users need to know if anything in the release has changed from the last time they saw it.
The COVID attendance publication has fortnightly releases at the time of writing, and the team have a good process for sign-off. This includes detailed information at each stage to confirm who has signed off, who has requested an action be taken, and who has carried out the action.
-Guidance for how to publish different types of statistics
-
-
-
-
-
Routes for publishing
-
-
Explore Education Statistics (EES) is the Department’s new dissemination platform, designed to make DFE’s published statistics and data easier to find, access, use and understand.
-
The platform moved into Public Beta in March 2020 and currently operates in parallel with existing statistics pages on gov.uk as publications are moved to the new platform.
-
More information on why EES was introduced and the functionality within the platform can be found in these slides.
-
-
All national, official and ad-hoc statistics should be published on EES
-
Management Information should be published on .gov.uk
-
Interactive dashboards built for particular user needs can be published separately, but should have a linked publication on either EES or GOV.UK
-
-
The following table outlines the key differences between publishing via EES and the old method via gov.uk:
-
-
-
-
-
-
-
-
Statistics collections on gov.uk
-
Explore Education Statistics
-
-
-
-
-
Statisticians request pdf and excel files are uploaded to release pages on gov.uk
-
Statisticians load csv data files on to the service and use them to build release pages
-
-
-
Release attachments are reviewed and approved via emails
-
Releases are reviewed and approved within the service
-
-
-
Release attachments are circulated for pre-release via email 24hours prior to publication date
-
Pre-release users are invited to preview releases within the service 24 hours prior to publication date
-
-
-
-
-
If you are unsure which route to use for publication please contact the HoP Office.
-
-
-
-
-
EES
-
-
All national, official and ad-hoc statistics should be published on EES. This section covers how to start publishing on EES and all the things you need to consider before publication. For more detailed guidance on how to use the platform, visit our EES guidance page
-
-
-
How to publish
-
-
-
-EES create release one pager
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Publication checklist
-
Before releasing statistics for the first time you may want to discuss the new process with key stakeholders and/or pre-release users to make them aware of the new service. You should also inform the Explore Statistics Mailbox and HoP Office teams.
-
Before you start creating a release in the platform you should have:
-
-
Announced the upcoming release via gov.uk
-
Sent metadata form to HoP
-
Contacted the BAU team so we can support you with your first release
-
Produced your tidy csv data files with appropriate disclosure control
title, summary sentence and ‘detail’ for the new page – you can include a link to previous releases if you want it to be the same
-
the link for your EES release – if you don’t have it you can update the ticket when the link is available
-
the link for the announcement
-
the link of any gov.uk collections it needs to be added to
-
an email with clearance from your deputy director
-
an email confirming communications are happy for it to go if it’s for stats that aren’t pre-announced or it’s an update to stats made after publication – so that they can prepare reactive lines
From [publication date], the Department for Education will be publishing the [publication title] on a new statistics site, Explore Education Statistics (EES).
Before you start creating your release you should have:
-
-
Announced the upcoming release via gov.uk
-
Sent metadata form to HoP
-
-
Before you publish a release you should have:
-
-
Checked that the data meets accessibility requirements, ideally as a .csv file, or an accessible Excel file where this is not possible
-
Written footnotes where appropriate
-
Passed the release for higher review (senior sign-off)
-
Sent the final files and sign-off email to the digital communications (gov.uk) team
-
-
-
-
-
-
Dashboards
-
-
Dashboards built to meet a particular user need can be another helpful tool to help users access and understand your data, and can be hosted outside of EES or gov.uk.
-
Guidance to follow when creating and publishing dashboards is available via the dashboards page.
-
-
-
How to publish
-
We recommend that teams use R Shiny to build and publish dashboards. Shiny applications are easily made accessible, and underlying code for Shiny apps can be published alongside for transparency.
-
For more information on publishing dashboards, including template code and how to get started, see our Public dashboards page.
-Guidance for how to use the tools available for statistics publishers
-
-
-
We also have a number of tools for statistics production teams to use to help them in their processes, they can be accessed at the links below when using a DfE device.
-
-
EES data screening app
-
This application allows you to screen your data files against the underlying data standards for Official and National statistical publications. If your file passes the checks it also gives you some basic options for exploring the data in the file.
-
-
-
-
-
\ No newline at end of file
diff --git a/resources/Survey_results.docx b/resources/Survey_results.docx
deleted file mode 100644
index 92f9f8c..0000000
Binary files a/resources/Survey_results.docx and /dev/null differ
diff --git a/search.json b/search.json
index f2af7de..40532ac 100644
--- a/search.json
+++ b/search.json
@@ -1,164 +1,213 @@
[
{
- "objectID": "writing-visualising/dashboards.html",
- "href": "writing-visualising/dashboards.html",
- "title": "Public dashboards",
+ "objectID": "CONTRIBUTING.html",
+ "href": "CONTRIBUTING.html",
+ "title": "Contributing to the Analyst’s Guide",
"section": "",
- "text": "Guidance for publishing public statistics dashboards"
+ "text": "Thank you for investing your time in contributing to the analyst’s guide! All contributions are very welcome - we want this to be a useful resource for the whole analytical community in DfE :sparkles:.\nRead our Code of Conduct to keep our community approachable and respectable.\nThe main branch of the site is protected and can only be updated via pull requests. All pull requests must be approved by a repository admin before merging.\nIn this guide you will get an overview of the contribution workflow from opening an issue, creating a PR, reviewing, and merging the PR.\n\n\nTo get an overview of the project, read the README. Here are some resources to help you get started with open source contributions:\n\nFinding ways to contribute to open source on GitHub\nSet up Git\nGitHub flow\nCollaborating with pull requests\n\n\n\nFor more information specific to how a quarto site works, and for examples of what is possible within a static quarto site, see the main Quarto website.\n\n\n\n\n\n\nIf you spot a problem with the site, search if an issue already exists. If a related issue doesn’t exist, you can open a new issue using a relevant issue form.\n\n\n\nScan through our existing issues to find one that interests you. If you find an issue to work on, you are welcome to open a PR with a fix.\n\n\n\n\nKey things to remember when making or proposing changes: - Where guidance already exists elsewhere it should be linked to rather than duplicated - When adding images, save them in the /images folder - When adding files for download, save them in the /resources folder - Don’t commit rendered .html files for pages - If your changes remove or edit a pre-existing anchor link, consider how that will be redirected for users who may have bookmarked it\n\n\nClick Edit this page at the bottom of the right hand table of contents of any page to make small changes such as a typo, sentence fix, or a broken link. This takes you to the .Qmd file where you can make your changes and create a pull request for a review.\n\n\n\n\nClone or fork the repository.\nOpen the repository in your editor of choice, e.g. R Studio.\nCreate a working branch and start with your changes!\nCommit and push the changes to your working branch.\n\n\n\n\n\nAll pull requests should be made against the main branch.\nWhen you’re finished with the changes, create a pull request, also known as a PR. - Don’t forget to link PR to issue if you are solving one.\nOnce you submit your PR, a repository admin will review your proposal. We may ask questions or request additional information. - We may ask for changes to be made before a PR can be merged, either using suggested changes or pull request comments. You can apply suggested changes directly through the UI. You can make any other changes and then commit them to your branch. - As you update your PR and apply changes, mark each conversation as resolved. - If you run into any merge issues, checkout this Git tutorial to help you resolve merge conflicts and other issues.\n\n\n\n\nIf you need any assistance at all, please contact statistics.development@education.gov.uk."
},
{
- "objectID": "writing-visualising/dashboards.html#user-needs",
- "href": "writing-visualising/dashboards.html#user-needs",
- "title": "Public dashboards",
- "section": "User needs",
- "text": "User needs\nWhen designing a government service, always start by learning about the people who will use it. If you do not understand who they are or what they need from your service, you cannot build the right thing.\nUnderstanding as much of the context as possible gives you the best chance of meeting users’ needs in a simple and cost effective way.\nThe real problem might not be the one you originally thought needed solving. Testing your assumptions early and often reduces the risk of building the wrong thing.\nServices designed around users and their needs:\n\nare more likely to be used\nhelp more people get the right outcome for them - and so achieve their policy intent\ncost less to operate by reducing time and money spent on resolving problems\n\nSee the service manual for more information on learning about your users and their needs."
+ "objectID": "CONTRIBUTING.html#new-contributor-guide",
+ "href": "CONTRIBUTING.html#new-contributor-guide",
+ "title": "Contributing to the Analyst’s Guide",
+ "section": "",
+ "text": "To get an overview of the project, read the README. Here are some resources to help you get started with open source contributions:\n\nFinding ways to contribute to open source on GitHub\nSet up Git\nGitHub flow\nCollaborating with pull requests\n\n\n\nFor more information specific to how a quarto site works, and for examples of what is possible within a static quarto site, see the main Quarto website.\n\n\n\n\n\n\nIf you spot a problem with the site, search if an issue already exists. If a related issue doesn’t exist, you can open a new issue using a relevant issue form.\n\n\n\nScan through our existing issues to find one that interests you. If you find an issue to work on, you are welcome to open a PR with a fix.\n\n\n\n\nKey things to remember when making or proposing changes: - Where guidance already exists elsewhere it should be linked to rather than duplicated - When adding images, save them in the /images folder - When adding files for download, save them in the /resources folder - Don’t commit rendered .html files for pages - If your changes remove or edit a pre-existing anchor link, consider how that will be redirected for users who may have bookmarked it\n\n\nClick Edit this page at the bottom of the right hand table of contents of any page to make small changes such as a typo, sentence fix, or a broken link. This takes you to the .Qmd file where you can make your changes and create a pull request for a review.\n\n\n\n\nClone or fork the repository.\nOpen the repository in your editor of choice, e.g. R Studio.\nCreate a working branch and start with your changes!\nCommit and push the changes to your working branch.\n\n\n\n\n\nAll pull requests should be made against the main branch.\nWhen you’re finished with the changes, create a pull request, also known as a PR. - Don’t forget to link PR to issue if you are solving one.\nOnce you submit your PR, a repository admin will review your proposal. We may ask questions or request additional information. - We may ask for changes to be made before a PR can be merged, either using suggested changes or pull request comments. You can apply suggested changes directly through the UI. You can make any other changes and then commit them to your branch. - As you update your PR and apply changes, mark each conversation as resolved. - If you run into any merge issues, checkout this Git tutorial to help you resolve merge conflicts and other issues."
},
{
- "objectID": "writing-visualising/dashboards.html#software-choices",
- "href": "writing-visualising/dashboards.html#software-choices",
- "title": "Public dashboards",
- "section": "Software choices",
- "text": "Software choices\nWhen you make a decision about technology, you’re making a significant investment. The choices you make will have a huge impact on your ability to create, iterate and operate the service in a sustainable way. We should be choosing tools and technology that let us create a high quality service in a cost effective way and minimise the cost of changing direction in future.\nFor public facing dashboards we recommended using R Shiny.\nWe feel that it best meets the service standards, it aligns with the departments RAP strategy for the use of R in in Official Statistics, and the general direction of the government analytical community.\nR Shiny is incredibly customisable, and ideal for providing everything in one place - interactivity, user customised printable PDFs (replacing old mail merge solutions), and ease of use on a phone!\nR Shiny takes advantage of transferable knowledge and shared resources that already exist to minimise the costs and maximise the shared benefits.\nWhile we recommend R Shiny, teams may use other tools if it can be justified against the Service Standard 11. Choose the right tools and technology.\nThe most common dashboard options available to us are:\n\nR Shiny\nPython Dash (or other Python-based alternatives)\nPowerBI\n\nSome considerations to think about when choosing the tooling / software for a dashboard:\n\nwill it be open source? (Service standards 12 and 13)\nwhat skills do your team already have, or are already developing?\ncan you reuse anything that someone else has already done?\ndo you have flexibility in formatting and styling?\nwill you be able to maintain it long term?\ndoes it give the flexibility required to meet accessibility requirements?\nwhat costs will it involve? (Consider learning and development time and courses as well as hosting)\nwill it allow you to develop automated testing and CI to QA our dashboards?\ndoes it align with the AF expectation to move towards R and Python for analysis?\n\nThe guidance on this page will focus on R Shiny. However, if you are using PowerBI you should also make use of the department’s PowerBI Dashboard standards, and aim to make use of reusable configurations where possible to ensure that duplication of effort across the department is minimised."
+ "objectID": "CONTRIBUTING.html#support",
+ "href": "CONTRIBUTING.html#support",
+ "title": "Contributing to the Analyst’s Guide",
+ "section": "",
+ "text": "If you need any assistance at all, please contact statistics.development@education.gov.uk."
},
{
- "objectID": "writing-visualising/dashboards.html#accessibility",
- "href": "writing-visualising/dashboards.html#accessibility",
- "title": "Public dashboards",
- "section": "Accessibility",
- "text": "Accessibility\nGovernment services must work for everyone who needs to use them. Public sector organisations have a legal duty to consider everyone’s needs when they’re designing and delivering services. Making a website or mobile app accessible means making sure it can be used by as many people as possible. This includes those with:\n\nimpaired vision\nmotor difficulties\ncognitive impairments or learning disabilities\ndeafness or impaired hearing\n\nAccessibility means more than putting things online. It means making your content and design clear and simple enough so that most people can use it without needing to adapt it, while supporting those who do need to adapt things.\nThe accessibility regulations came into force for public sector bodies on 23 September 2018. They say you must make your website or mobile app more accessible by making it ‘perceivable, operable, understandable and robust’. The full name of the accessibility regulations is the Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018. The accessibility regulations build on your existing obligations to people who have a disability under the Equality Act 2010 (or the Disability Discrimination Act 1995 in Northern Ireland).\nThese regulations apply to all dashboards created for Official statistics. All dashboards should:\n\nhave an accessibility statement\nfollow the guidance below for testing against WCAG 2.1 AA accessibility standard\nuse open data that is already available via EES\n\nAnalysts should familiarise themselves with the gov.uk guidance on service accessibility"
+ "objectID": "statistics-production/examples.html",
+ "href": "statistics-production/examples.html",
+ "title": "Good examples in EES",
+ "section": "",
+ "text": "Examples of good practice in EES, from file names to content"
},
{
- "objectID": "writing-visualising/dashboards.html#governance",
- "href": "writing-visualising/dashboards.html#governance",
- "title": "Public dashboards",
- "section": "Governance",
- "text": "Governance\nAll teams developing dashboards to accompany Official Statistics should contact the Statistics Development Team for advice on the level of approval required as there may be times when HoP approval is needed in addition to team/unit leader approval. Approvals should be sent to the Statistics Development Team when asking for a new dashboard to be hosted on shinyapps.io.\nThe standards for dashboards, and department strategy are maintained by the Central Statistics Standards Unit and governed by the Statistics Leadership Group, which is made up of all senior statisticians owning Official Statistics publications in the department.\nRegular reflection on how teams are doing and finding ways to improve is an important part of good governance. We recommend that Senior Statisticians responsible for dashboards ensure that they are regularly reviewed, covering:\n\ntesting and quality assurance\naccessibility\nperformance against user needs\noverall coherence with central standards and strategy found on this page"
+ "objectID": "statistics-production/examples.html#files",
+ "href": "statistics-production/examples.html#files",
+ "title": "Good examples in EES",
+ "section": "Files",
+ "text": "Files\nFiles can be downloaded from multiple areas across the platform and often users will judge their contents based on the name alone. For display names, it’s important we make it as clear as possible to users as to what each file contains without cluttering the title with information that is already available from the page around it, saving detailed coverage for the data guidance.\n\n\nFile names\n\nFull guidance for naming files.\nThe Education and training statistics for the UK release has some good examples of file names which clearly explain what is in the file, while falling well under 35-50 characters in length, e.g.:\n\nuk_schools.csv\nuk_expenditure.csv\nuk_pupils.csv\n\nThe file names do not have time periods in them, making them easier for users to make use of newer versions in future\n\n\n\nSubject names\n\nFull guidance for naming subject files.\nThe School Workforce Census applies this guidance well, with short and simple titles that make sense to a non-expert, e.g.:\n\nTeacher pay\nTeacher retention\nSubjects taught\n\nThese titles are not cluttered with geographies and time periods, which automatically populate in the subject metadata.\n\n\n\nSupporting files\n\nFull guidance for using supporting files in EES.\nThe Initial Teacher Training performance profiles publication is one example of needing to upload a file outside of the regular data/metadata structure. Their provider-level tables also contain national and regional data, which is not supported in EES at the time of writing. The file has a clear title, and data guidance clearly explains what is in the file."
},
{
- "objectID": "writing-visualising/dashboards.html#iterative-development",
- "href": "writing-visualising/dashboards.html#iterative-development",
- "title": "Public dashboards",
- "section": "Iterative development",
- "text": "Iterative development\nUsing agile methods means getting your service in front of real users as soon as possible. Then observing and generating data on how they use it and iterating the service based on what you’ve learned. Because you’re not specifying everything up front before you’ve developed an understanding of what users need, agile methods reduce the risk of delivering the wrong thing.\nTeams should aim to get a working version of their dashboard out to users as soon as possible to prototype the design and content.\nIteration isn’t just for the early stages of a service’s development and services are never ‘finished’. Using agile methods means getting real people using your service as early as possible. Then making improvements throughout the lifetime of the service.\nMaking improvements means more than doing basic maintenance like fixing bugs in code, deploying security patches and keeping call centre scripts up to date. If that’s all you do, you’ll be fixing symptoms rather than underlying problems. And over time, the service will stop meeting user needs.\nContinuous improvement means you can respond to changes in user needs, technology or government policy throughout the lifetime of the service. So rather than having to be replaced, the service stays relevant until it’s ready to be retired."
+ "objectID": "statistics-production/examples.html#data-guidance",
+ "href": "statistics-production/examples.html#data-guidance",
+ "title": "Good examples in EES",
+ "section": "Data guidance",
+ "text": "Data guidance\nThe public data guidance is a key element of your release, to help direct users to the right file to download or build tables with. It is frequently highlighted by users of our stats as a really helpful element of EES. You will be prompted to fill in data guidance in the release checklist. Any release that does not have a complete public data guidance will not be blocked from approval and publishing.\n\n\nPublic data guidance\n\nFull guidance for public data guidance.\nThe School Workforce Census, along with having sensible brief file names, also has great examples of useful data guidance, which explain what is in each file without overloading the user with information:\n\n\n\n\nPublic data guidance - overview\n\nFull guidance for public data guidance.\nThe NEET annual brief is a good example of a public metadata overview section done well. It follows the EES template, goes into just the right amount of detail, and is easy to understand.\n\nNEET data guidance on EES."
},
{
- "objectID": "writing-visualising/dashboards.html#code-testing",
- "href": "writing-visualising/dashboards.html#code-testing",
- "title": "Public dashboards",
- "section": "Code testing",
- "text": "Code testing\nTesting is a way to capture desired behaviour of your code, in such a way that you can automatically verify that it keeps working the way you expect over time. It is essential for making sure that code works the way that you intend it to, and keeps working even after you make changes to the code so that your users have access to a stable service. You need to test your service regularly as part of quality assurance (QA) to make sure that it:\n\nis easy to use for anyone who needs to use it, regardless of the device they’re using\nis stable, secure and works quickly, regardless of how many people need to use it\ncan be iterated quickly to meet changes to user needs or the political environment\n\nTests can come in a variety of shapes and sizes, good starting points for analysts new to testing are the Duck Book, and DfE good code practice.\nYou should aim to automate as much of your testing as possible and run your test suite as part of continuous integration (where your tests form part of your codebase). By testing your code automatically every time you make a change, you’ll be able to find defects more quickly. Getting fast feedback means you can respond to any problems quickly and make changes when you need to. You can also spot defects before they develop into bigger problems that are more complicated and expensive to fix.\nFor dashboards created in R Shiny, we strongly recommend a mix of unit tests and UI tests (integration), run using GitHub Actions for continuous integration (CI). All applications should have an appropriate level of test coverage before being published. More information and details on how to get started with this can be found in the testing R Shiny section below."
+ "objectID": "statistics-production/examples.html#charts-and-tables",
+ "href": "statistics-production/examples.html#charts-and-tables",
+ "title": "Good examples in EES",
+ "section": "Charts and tables",
+ "text": "Charts and tables\nCharts and tables are key ways to highlight important information and stories in your release. They should be clear, contain few footnotes that are essential to interpreting the chart or table, and have good descriptive alternative text so users with screen readers can understand what is being visualised.\n\n\nOther infographics\n\nFull guidance on using charts built outside of EES.\nThe Further Education: outcome based success measures release contains infographics that cannot be built in EES. They are saved as SVGs so render clearly on any device, and follows the same colour palette as the rest of the release.\n\n\n\n\nCharts and footnotes\n\nFull guidance on creating charts, and guidance on creating footnotes.\nThe Widening participation in higher education publication has a good example of a chart built with a long timeseries in the chart builder, with sparing use of footnotes. For example below, a line in the chart indicates a point where comparisons cannot be made, and a key caveat around the comparability of years is included as a footnote, which is needed to interpret the data and chart accurately.\n\n\n\n\nAccessibility and alt-text\n\nGood alt-text descriptions in charts will not just repeat the title of the chart, but instead describe the type of chart, what the data coverage is, and what trends can be seen in the chart to a user accessing the page with a screenreader.\nThe school workforce census has a good example of alt-text for one of their charts. The associated alt-text for the chart below is: “Line chart showing the percentage of all teachers taking absence across all state funded schools in England between the academic years 2014/15 and 2018/19. The chart shows a fall from 55.7% to 54.0% in this period.”\n\n\n\n\nFeatured tables\n\nFull guidance on creating featured tables.\nGood featured tables have short, easy to understand titles, and point users to commonly requested tables or other tables that might be interesting to a wide range of users. For example, the pupil absence in schools: autumn term release has a good example of a clear title, with further information (including date and geographic coverage) left in the description. Leaving these out of the main title makes it much easier for users to navigate through multiple featured tables:"
},
{
- "objectID": "writing-visualising/dashboards.html#assessing-engagement",
- "href": "writing-visualising/dashboards.html#assessing-engagement",
- "title": "Public dashboards",
- "section": "Assessing engagement",
- "text": "Assessing engagement\nWork out what success looks like for your service and identify metrics which will tell you what’s working and what can be improved, combined with user research. For dashboards this will likely be things like the number of users and interactions with what you have created.\nDefining what “good” looks like and identifying appropriate metrics means that you’ll know whether the service is solving the problem that it’s meant to solve.\nCollecting the right engagement data means you’ll be alerted to potential problems with your service. And when you make a change to the service, you’ll be able to tell whether it had the effect you expected. In practice this will often mean setting up analytics and feedback surveys for dashboards and monitoring the data you get back.\nAt a minumum you should be requesting feedback from users via a survey hosted on the dashboard and reviewing this on a regular basis. An example of this kind of feedback survey is the beta banner survey on Explore Education Statistics.\n\nGoogle Analytics is a free service that collects information on who visits your webpage and how they interact with it. You can set up basic Google Analytics for your published dashboard in a few simple steps outlined in this article: Add Google Analytics to a Shiny app and view a more complex example in this example file. If you’re planning to publish a dashboard, or to set up Google Analytics for a published dashboard, please contact statistics.development@education.gov.uk."
+ "objectID": "statistics-production/examples.html#content",
+ "href": "statistics-production/examples.html#content",
+ "title": "Good examples in EES",
+ "section": "Content",
+ "text": "Content\nClear, concise content is required to direct your users to the right place, and keep them engaged with your release. Our content guidance page contains a raft of great advice on how to structure your release and write for the general public.\n\n\nHeadlines\n\nFull guidance on creating a headline section\nThe Education, health and care plans release has a good example of a headlines section. There are a sensible number of key statistics, and the summary below gives an overview of trends without diving into the numbers and overwhelming the user.\n\nThe Exclusions release has a good example of custom explanations under key stat tiles to show users exactly what the numbers are describing:\n\n\n\n\nAccordion content\n\nFull guidance on writing content for accordions\nThe parental responsibility measures release has good examples of writing for the public in accordion content. Content in each accordion is short and follows the “pyramid” principle of having essential information at the top, summarising the trend, then going into detail at the bottom.\n\n\n\nActive subheadings\n\nFull guidance on using active headings and titles\nThe EHCP release has some great examples of active subheadings within accordions, which explain the overall trends without overloading users with detail.\n\n\n\nGlossary links\n\nThe summary of the CIN / CLA outcomes release has examples of this in action.\n\n\n\nMethodology\n\nFull guidance on creating a methodology in EES.\nGood methodologies will broadly cover topics in our recommended methodology template. The Graduate outcomes (LEO) methodology broadly follows this outline, making use of effective formatting in EES to help users navigate through accordions, although definitions can be moved to the EES glossary.\nThe Key stage 4 destinations methodology makes good use of annexes to show changes to their methodology over time."
},
{
- "objectID": "writing-visualising/dashboards.html#peer-review",
- "href": "writing-visualising/dashboards.html#peer-review",
- "title": "Public dashboards",
- "section": "Peer review",
- "text": "Peer review\nPeer review is a quality assurance activity, where an analyst other than the original author, views and tests the usage of a product or specific piece of code. This allows a fresh pair of eyes to take a look at your work. It validates that you have taken a correct approach and may highlight errors. This constructive feedback helps you to improve the quality. It provides confidence in your work, and ensures that it is fit for purpose.\n\nDashboards must always be peer reviewed within the team they are created.\nDashboards should also be peer reviewed by analysts outside of the subject area of the team.\n\nThe Central Statistics Unit has a number of analysts experience with R Shiny dashboards and are happy to review any dashboards created, contact statistcs.development@education.gov.uk if you’re interested in this. For more guidance on how to peer review, see the peer review section of the Duck Book."
+ "objectID": "statistics-production/examples.html#approvals-and-amendments",
+ "href": "statistics-production/examples.html#approvals-and-amendments",
+ "title": "Good examples in EES",
+ "section": "Approvals and amendments",
+ "text": "Approvals and amendments\nKeeping track of processes for approving and amending releases is crucial for transparency. Internally, there should be clear paper trail of who has signed off the release, and externally, users need to know if anything in the release has changed from the last time they saw it.\n\n\nPublic amendment notes\n\nFull guidance for creating public amendment notes.\nThe Pupil absence in England - Autumn and Spring terms publication has some good examples of release notes.\nThese notes clearly explain to the user what has changed, which files were affected and at what level:\n\n\n\n\nInternal release status notes\n\nFull guidance for creating public amendment notes.\nThe COVID attendance publication has fortnightly releases at the time of writing, and the team have a good process for sign-off. This includes detailed information at each stage to confirm who has signed off, who has requested an action be taken, and who has carried out the action."
},
{
- "objectID": "writing-visualising/dashboards.html#internal-only-dashboards",
- "href": "writing-visualising/dashboards.html#internal-only-dashboards",
- "title": "Public dashboards",
- "section": "Internal only dashboards",
- "text": "Internal only dashboards\nIt’s possible to publish dashboards that are only accessible to those on DfE kit, to do this you will need to publish via the departments RSConnect servers.\nYou will need:\n\nA finished app, in line with the guidance on this page\nThe code to be in a Git repository in the dfe-gov-uk Azure DevOps space\n\nTo publish the app, you’ll need to set up a pipeline in Azure DevOps, guidance for how to do this can be found in the R community teams area.\nAccess to applications on RSConnect are locked down by default, once the pipeline is set up and you’ve deployed the app you’ll need to request for its access to be opened up by using an RSConnect request on ServiceNow.\nIf you’re running the app from an internal database, you’ll need to contact the database owner to set up a local login, and then store those as variables against your specific app in rsconnect. You can raise a request to do this via ServiceNow and selecting ‘Change app variables’."
+ "objectID": "statistics-production/ees.html",
+ "href": "statistics-production/ees.html",
+ "title": "Explore Education Statistics",
+ "section": "",
+ "text": "Guidance for how to use the features in the Explore Education Statistics platform\nExplore Education Statistics (EES) is the Department’s official statistics dissemination platform, designed to make DFE’s published statistics and data easier to find, access, use and understand.\nThe EES platform consists of two applications:\nBoth applications were released to Minimal Viable Product (MVP) standard in March 2020, we will be iterating and improving the functionality based on user feedback throughout 2020 and 2021.\nMaintenance and use of the platform is supported by the Statistics Development Team."
},
{
- "objectID": "writing-visualising/dashboards.html#public-dashboards",
- "href": "writing-visualising/dashboards.html#public-dashboards",
- "title": "Public dashboards",
- "section": "Public dashboards",
- "text": "Public dashboards\nThe majority of dashboards made to support and augment our Official Statistics will be public facing. For public facing shiny apps you should publish via shinyapps.io. The statistics development team manage a subscription for this and can help you get set up.\nYou will need:\n\nA finished app that meets the accessibility and styling standards (see our Dashboard procedure checklist)\nCode in the dfe-analytical-services GitHub repo\nApproval from your DD\nIf the data underlying the dashboard is currently unpublished, you will need to create dummy data to use in GitHub until the data becomes published (see dummy data guidance section).\n\nTo set up a new app, send the above to statistics.development@education.gov.uk. If your code is not yet hosted in the dfe-analytical-services area you can request for the repository to be moved at the same time as sending approvals.\n\n\nDummy data\n\nWhen creating a public dashboard, all code and data will be hosted in the dfe-analytical-services GitHub repo. This is a public repo, meaning anything stored in here is publicly available. Therefore if you are creating a dashboard with previously unpublished data, you should provide dummy data for the GitHub repo, and only add the real data on your publication date.\nYour dummy data should:\n\nUse the exact same structure and variable names as your real data\nUse random values in place of real values (one example of how to do this is using the rnorm() function)\nSimulate the types of suppression, missing data or anomalies seen in your real data, to ensure the dashboard can account for these.\n\n\n\n\nTesting with unpublished data\n\nWhile you must use dummy data in your GitHub repo, it is understandable that we should test the dashboard works with the real data before it goes live.\nThis can be done using the .gitignore file alongside the datafiles_log.csv and commit hooks explained in the Stopping accidental data uploads guidance. You can view the example files our template repository.\nThe .gitignore file is a plain text file that tells git to ignore specified files in commits and pushes to the repo. Therefore, the first step when wanting to test unpublished data in a dashboard is to add the unpublished data file names to the .gitignore file.\nAdding the file name alone will ensure it is ignored no matter where in the project area you save it (i.e. in any folder). Once this is done, you can add your unpublished data file to your local area, run the app locally, and make edits/commits without uploading the data to Github.\nThis .GitIgnore guidance page has great guidance on how you can utilize wildcards to capture all the files you might want to ignore.\nIf you have any questions on this process please do contact us at statistics.development@education.gov.uk."
+ "objectID": "statistics-production/ees.html#environments",
+ "href": "statistics-production/ees.html#environments",
+ "title": "Explore Education Statistics",
+ "section": "Environments",
+ "text": "Environments\nAs stated above, EES consists of two parts. We also have four versions (environments) of EES, the banner for the admin part of each environment will inform you which environment you are on, this is also colour coded:\n\nDevelopment - Green - where changes are first merged in, and often the first time different pieces of work from different developers will interact properly\nTest - Pink - where our developers can carry out manual testing of any new features to make sure things work as expected\nPre-production - Yellow - a sandbox area for analysts to carry out functionality testing\n\nAdmin pre-production - https://admin.pre-production.explore-education-statistics.service.gov.uk\nPublic pre-production - https://pre-production.explore-education-statistics.service.gov.uk\n\nProduction - Red - the real service, anyone creating real releases that they intend to publish to the public should be using this environment\n\nAdmin production - https://admin.explore-education-statistics.service.gov.uk\nPublic production - https://explore-education-statistics.service.gov.uk\nThere is no overlap between the environments and content created on one cannot be moved to any other.\n\nIf you are bookmarking links, please be careful to bookmark the links below exactly as they are shown. Often when signing in you will be redirected via other url’s as a part of the authentication process, and bookmarking those may lead to errors.\n\nTeams can use the pre-production environment to familiarise themselves with the platform and test out what is possible. The core functionality across the environments is identical except for new changes, which are deployed through the different environments before they make it to production.\nOn the pre-production environment, all analysts have full permissions to create publications and releases, and can see everything else that other analysts are making. This is unique to the pre-production environment, on the production environment analysts will only have access to releases that they have been granted specific access to.\nWhen accessing the pre-production environment you may be asked for a username and password, these are as follows: dfe, dataresearch.\n\nThe pre-production environment is not suitable for unpublished data. Unpublished data should only be uploaded to the production environment.\n\n\nGetting access to admin\nAccess to the production and pre-production admin services is limited to DfE AD accounts only and users have to have been invited to the service by either the Statistics Development Team (for full access) or an existing user (for pre-release access). Invites to the service are sent out via email using gov.uk Notify.\nTo be invited to the service for full access teams need to email the Explore Statistics Mailbox, stating who needs access, what permissions they require (analyst or approver), and for which publications these apply. This email should be sent by the Team Leader, or accompanied with the relevant Team Leader’s approval. Once access has been granted you will receive an email inviting you to use the platform.\n\nJobshare emails can not be used to access EES. Specific personal emails should be used instead, in the same way as you sign in to windows on your machine."
},
{
- "objectID": "writing-visualising/dashboards.html#how-to-start",
- "href": "writing-visualising/dashboards.html#how-to-start",
- "title": "Public dashboards",
- "section": "How to start",
- "text": "How to start\nThere are a lot of resources already available to support you when working with R Shiny, if you’re new, then this blog post provides a gentle introduction.\nOur template shiny app repository is a useful starting point for all public facing dashboards in the department. Please see the dashboard template section for guidance on what the dashboard template includes and how to use it.\nFor more advanced shiny knowledge it’s worth taking a look at the guide to engineering production-grade shiny apps.\nFinally, you should seek to make use of the community that is already out there, see what others are doing, ask them questions and for advice on any decisions or problems that you’re facing, and share what it is that you’re doing. - The Statistics Development Team are experienced with R Shiny and happy to help or offer advice, and there is the DfE R community on teams. - There is a bank of tips and tricks stored in the DfE R teams area too. - Going beyond DfE there’s a wealth of resources and communities online, including Stack Overflow discussions, cross government slack channels (e.g. #R and #Shiny on govdatascience.slack.com) and even tweets about R Shiny on twitter.\n\n\nDashboard procedure checklist\n\nThis checklist outlines the standard procedure for teams who are wishing to produce a public R shiny dashboard.\nGetting set up:\n\nCreate an account on GitHub\nAsk the Statistics Deveopment Team to create you a repository in the DfE analytical services area, providing the name of the dashboard and the GitHub accounts of anyone who will be contributing to the code. You should aim to have two analysts working on the code development and a line manager for review purposes. Further colleagues with review responsibilities (policy colleagues, G6 and above, etc.) can be given access to a demo-site, rather than the repository (see guidance for this below in ‘Setting up a development/demo dashboard area’).\nClone the repo to your device so you can develop your code. Open the repo page in GitHub, click the green ‘Code’ button, and copy the URL it provides. Then, open R Studio on your device, click file > new project > version control > git, paste the repository URL you copied from GitHub, give your local project area a name, and choose where to save it (i.e. on your computer’s C: drive, outside of the OneDrive-synced folders).\n\nOnce you’re set up, there are certain parts of the code you need to update:\n\nIn the global.R script, update all of the site URL’s and EES publication names to your own.\nIn the ui.R script, update the tags$title(), the information in the meta_general(), and the secondary text in shinyGovstyle::header().\nGot to the .github > workflows folder, and open the deploy-shiny.yaml file. At the bottom, update the appName in rsconnect::deployApp() - this is what will appear in the URLs (i.e. must align with the links in global).\nUpdate the ‘Support and feedback’ tab (in R > standard_panels.R) with your teams information. We also recommend creating a feedback form for users of your dashboard, and adding a link to that on this page.\nUpdate the README.md file (you can do this easily directly in GitHub). This is the file that renders below your repo in GitHub for users to see.\nBegin adding your own dashboard content. If you copy and paste any parts of the code (i.e. to create new tabs) you must change all of the IDs so there are no repeated IDs in the UI, otherwise the app will run with no UI elements. You should add UI and unit tests as you develop your code as a form of automated QA (see our guidance on UI tests and guidance on unit tests).\n\nYou must contact the statistics development team for the following:\n\nTo add the shinyapps.io secret and token to your GitHub repo, therefore enabling the app to be hosted on shinyapps.io.\nTo create an area for your team in Google Analytics, to track the user analytics of your dashboard.\n\nSetting up a development/demo dashboard area:\n\nWhile developing your dashboard, you may want a private, demo-version to share with policy or senior colleagues for review and feedback. This version must use either published data or dummy data and can not use unpublished data, since this cannot be uploaded to GitHub until the day of publication (see our dummy data guidance for public dashboards).\nEnsure that prior to contacting the statistics development team, you have updated all of the URL’s and other items listed above.\nYou must contact the Statistics Deveopment Team to add the shinyapps.io secret and token to your GitHub repo, therefore enabling the app to be hosted on shinyapps.io. Once this is done you will have a browser link you can use to access the dashboard. We can make this private such that there is a list of approved viewers who must log in to view the dashboard - please provide the email addresses of any colleagues who you wish to have access to the private version during development.\n\nYou must have done the following before a dashboard can be published (the statistics development team must review and approve that these have been met):\n\nAccessibility testing the dashboard and updating the accessibility statement. The accessibility testing guidance later in this section outlines how teams can do this.\nYou should test how your dashboard appears and performs on mobile devices. you can do this by opening your dashboard in chrome/edge, right clicking anywhere on the page and selecting ‘inspect’. you will then see a new panel on the right hand side of your screen. To see your dashboard how a mobile user would, click the mobile/tablet button (see the image below).\n\n\n\nSetting up UI and unit tests (UI tests are a part of the automated QA process and will run via GitHub actions each time you create a pull request). See our guidance on UI tests and guidance on unit tests.\nPerformance testing your dashboard. See our guidance on performance testing.\nThe underlying data for the dashboard must still be uploaded to EES in the tidy data format to be used in the table tool (check this using our data screener)\nDecide where you are going to provide the public the link to your dashboard. You can easily add a link to your EES publications. If you have a draft release that is going out on the same day as the dashboard, you can add the link into your EES draft while the shinyapps.io page is still private. This is because the link will not change when the dashboard is made public, the log-in screen will simply be removed."
+ "objectID": "statistics-production/ees.html#roles-and-permissions",
+ "href": "statistics-production/ees.html#roles-and-permissions",
+ "title": "Explore Education Statistics",
+ "section": "Roles and permissions",
+ "text": "Roles and permissions\nThe Statistics Development Team are responsible for setting up and maintaining user permissions during the beta phase. Change requests will be monitored via their main mailbox - explore.statistics@education.gov.uk.\nThe following roles exist within EES admin and are assigned to a specific release:\n\nPre-release\nAny user invited for pre-release is given a pre-release user role. During the hours of pre-release they can:\n\nPreview the release page, including downloading files\nPreview the table tool page for that release\n\n\n\n\nAnalyst - contributor\nAny analyst working on a release within EES admin will have the contributor role. They can:\n\nEdit release details\nUpload data and other files\nCreate footnotes, datablocks and charts\nEdit and comment on release content\nInvite PRA users\nMove the release status between draft and ready for higher review\n\n\n\n\nAnalyst - approver\nThe responsible statistician for a statistics release will have the approver role, allowing them to sign off releases for publication. This will usually be the responsible G5 or G6 for the statistics publication. They can do everything a contributor can, as well as:\n\nMove the release status between draft, ready for review and approved\nSchedule the release for publication (either immediately or for a specific date)\n\n\n\nThere is also a publication owner role, which is assigned at publication level to a users account:\n\nPublication owner\nThis permissions level gives publication owners control over their publications. They can do anything a contributor can for all releases within the publication. On top of this the publication owner also has access to:\n\nManage publication level details\nCreate new releases within a publication\nCreate an amendment of the latest published release within a publication\nCancel an amendment before it is published\n\n\n\n\nBAU\nThe administrative role:\n\nThis role is assigned to a user and gives full access including administrative tools, typically only used by the Statistics Development Team and EES developers.\n\n\nSee below for a diagram of the responsibilities of each role as part of the publication process in EES:"
},
{
- "objectID": "writing-visualising/dashboards.html#standards-to-follow",
- "href": "writing-visualising/dashboards.html#standards-to-follow",
- "title": "Public dashboards",
- "section": "Standards to follow",
- "text": "Standards to follow\nWe expect all dashboards to follow a minimum set of standards to ensure coherence between our products and a minimum standard of quality for our end users.\nThese standards are constantly evolving, and all feedback and contributions are welcome, contact us at statistics.development@education.gov.uk.\n\n\nAccessibility testing\n\nIn line with recent legislation for public sector websites, all dashboards need to meet the latest Web Content Accessibility Guidance.\nAs a minimum we expect all dashboards to be checked using the two tools below (lighthouse and shinya11y) and have an accessibility statement before being published.\nYour statement should be written in line with the accessibility statement guidance, and you can make use of a template accessibility statement provided by .gov.uk.\nLighthouse in Google Chrome is an easy way to quickly rate your accessibility. Open your app in Chrome, right click anywhere on the page and select “Inspect”. From there, navigate to “Lighthouse” in the top grey bar, and click “Generate report”. This generates scores for accessibility, best practices and SEO for your application or web page:\n\nTo complement this, we recommend also using the shinya11y package to look at accessibility across your dashboard. Simply install and load the package, then include use_tota11y() at the top of your UI script. This brings up an interface that helps you examine different accessibility aspects in your app, like colour contrast, alt-text and what screen readers will detect:\n\nAutomated tools can’t check for everything and there’s no substitute for giving your dashboard a manual check. You should consider what users need from your dashboard and ensure that that information is accessible to all. For example, can someone using a screen reader get the same information as you can see on the screen from a downloadable csv or from the alt text that you have provided? Everything that you can see, you should also be able to read with a screen reader. Use the Edge narrator tool to test out how your dashboard works in practice.\nOne example of a dashboard for Official statistics that meets these regulations is the SCAP LA school places scorecards app. Their accessibility statement is clearly labelled, explains what checks have been carried out, what the known limitations are and the plans in place to fix them:\n\n\n\n\n\nStyling\n\n\nAll dashboards should have a link to the source code, and information about who to contact for feedback/issues. You should be familiar with and follow the gov.uk style guide as appropriate.\nTeams should pick from the gov.uk design system colours when creating dashboards. The style sheet in our template repository makes use of these colours, though this can be extended if needed to suit your needs. Be careful when choosing colours, and make sure that colours have sufficient contrast to be placed next to each other to meet WCAG AA standards using the colour contrast checker.\nCharts using ggplot2 and plotly can take custom palettes using the scale_fill_manual() function, e.g. below:\n# Create colour palette using recommended colours (based on gov.uk design system)\n\ndfe_colours <- c(\n \"#12436D\", #`blue`\n \"#F46A25\",#`orange`\n \"#801650\",#`maroon`\n \"#28A197\" #`turquoise`\n )\n\n# Create chart using palette\n\nggplot(aes(y=value, x=\"\",\n fill = factor(rating),\n text = paste(rating, \": \", value, \"%\"))) +\n geom_bar(stat=\"identity\", position = position_fill(reverse = TRUE))+\n scale_fill_manual(values = dfe_colours)\n\nFurther examples are given on the Using Explore Education Statistics guidance page.\nYou can use in-line CSS to make these style changes, or make edits to the style sheet. Editing the style sheet is a neater solution if you want to apply changes across your app, and keeps your UI code clean. You can find out how styles are configured by running your app, right clicking and selecting “Inspect” to view the element you wish to change. You can then find these lines in the CSS stylesheet and change a number of things including fonts, font colours and background colours.\nThe SCAP LA school places scorecards app is one example of a public facing dashboard that meets these styling requirements.\n\n\n\n\nTesting R Shiny\n\nTo ensure that they are reliable, dashboards should have an appropriate amount of automated tests before they can be published. We recommend using a mix of UI and unit tests, but the number and type of tests that you need to run will depend on the content of your application.\nFurther guidance on setting up testing can be found in the DfE good code practice guide. Also see our guidance on testing R code on the learning resources page.\n\n\nPerformance testing\n\nPerformance profiling represents a chance to breakdown how different elements within your dashboard perform relative to each other. This consists of running through a set of actions on a local run of your dashboard and timing each one. A useful tool for performing these tests is profvis, which carries out all the timings and visualisation of the results for you, whilst all you have to do is run the app and step through the actions that you want to profile.\nThe basic steps are as follows.\nWith your dashboard repository open in RStudio, run the following commands from the R console\ninstall.packages(\"profvis\")\nprofvis(shiny::runApp())\nThen go through a set of interactions in your dashboard (e.g. navigate to the dashboard page, step through changing the selected choices in each input field, switch between any different tabs and cycle through the relevant inputs again). Once you’ve performed a representative range of actions in the dashboard, close the window and go back to the RStudio window.\nAfter several seconds (depending on the range of actions you performed), a visualisation will appear in the main RStudio editor window showing the results of the profiling.\nThe profvis results are shown in the flame profiling chart. The top half of the chart shows any processing time measurements above 10ms for each line of code. It can be useful to focus in on lines that have measured times above 10ms by selecting “Hide lines of code with zero time” from the options menu in the top right.\n\nLines with timings of greater than about 5-100ms may warrant further investigation to see if they warrant some optimization. Common routes to optimising code are:\n\navoid any data processing in server.R, e.g. avoid using the summarise() function;\nuse reactive() elements to minimise repeated calls of given functions;\nuse memoise() to cache results of more intensive functions (e.g. ggplot) to minimise repeated calls.\n\nThe documentation for the profvis package can be found here: Profvis documentation\nWhilst profvis can help identify any bottlenecks, this will ideally be complemented in the future by full load testing of dashboards, whereby the behaviour of dashboards under real-life high demand scenarios can be tested. However, this type of testing is unavailable whilst our applications are hosted on ShinyApps. We will offer support on load testing once we move dashboards on to our own server systems.\n\n\n\nUI Tests\n\nUser interface (UI) tests should be used for all apps. These tests should check that:\n\nYour app loads up correctly\nContent appears as expected when you switch between tabs\nContent updates as expected when you change inputs\n\nThe shinytest package is a really easy way to create these tests on an RShiny app. Simply load in the package and run shinytest::recordTest() to open up the testing environment to get started.\nEach test should produce one “expected results” JSON file. You can track particular elements of your app by holding CTRL and clicking on an element in the testing window.\nFor example, the SCAP LA school places scorecards app has tests that check that the app functions as expected when different combinations of filters are selected.\nEach test is separated out by comments, clearly stating what is being tested. - app\\(setInputs()** tells the UI test to select an input in the app - **app\\)snapshot() tells the UI test which outputs to capture (i.e. which outputs you’d expect to change and want to test)\n\nThe expected outputs are saved in a UI_tests-expected folder, and every time you run a UI test, the outputs will be compared to these expected snapshots.\nAn example of a basic shiny test to check that your app loads up correctly can be found in our shiny template repo.\n\n\n\nUnit Tests\n\nUnit tests should be used for apps that contain custom functions created from scratch, or that rely on existing functions being combined to interact in a certain way.\nUnit testing checks, for a set of defined variables, that the output is always the same when you run your custom function/s. This ensures that if slight tweaks are made to your function, or functions within packages you are using are updated, that your custom function still works.\nFor example, our QA app has many custom functions that we create to screen files before upload to EES. We have a series of dummy files that we have created to either pass or fail certain tests. Unit tests are written to check that these files always pass or fail as expected.\n\n\n\nTests and deployment\n\nBoth UI and unit tests need to be added to your app’s deployment pipeline. This ensures that the app will not be published if any of the tests fail.\nWe recommend using Github Actions to deploy the latest code whenever a push to your master branch occurs - this ensures that the published version of your app stays up to date.\nYou should also use GitHub Actions to run the automated tests for your app, which we recommend are done on at least every pull request.\nIf you’ve started from our template repository then all of this will be mostly set up for you to tweak and expand on, but if you haven’t then you’ll need to add the yaml files from the .github/workflows folder to your repository.\nGitHub actions are already well documented and their own guidance should be able to walk you through anything you need. That being said, if there’s anything else you’d like to see here let us know.\n\n\n\n\n\nConnecting to databases\n\nWe don’t yet have a database server set up that can be accessed by public facing dashboards, though we are working to put this in place. In the meantime, there a few alternative options for storing data that the dashboard can run off.\n\nStore the data within the repo (e.g. CSV files in a /data folder). Note that only published data should be stored in a repo. If your working on a dashboard that uses unpublished data then please see the dummy data section for guidance.\nUse Google sheets\nUse Dropbox\n\nIf you are running an internal-only app then you can connect to our internal SQL servers, information on how to do this is in the R community teams area.\n\n\n\nSecure variables\n\nSee our Git page for guidance on storing secure variables in repositories."
+ "objectID": "statistics-production/ees.html#requesting-a-new-publication",
+ "href": "statistics-production/ees.html#requesting-a-new-publication",
+ "title": "Explore Education Statistics",
+ "section": "Requesting a new publication",
+ "text": "Requesting a new publication\nIf you have not published on EES before, or if you’re creating a brand new publication, you will need to contact us to create the publication for you before you can get started.\nThe hierarchy of content within EES is as follows -\n\nPublications are organised into topics and themes (as shown on the EES Find Statistics page), and then within each publication there are releases - where the latest release includes the latest statistics for that publication. For example:\n\n\n\nLevel\nExample\n\n\n\n\nTheme\nPupils and Schools\n\n\nTopic\nPupil absence\n\n\nPublication\nPupil absence in schools in England\n\n\nRelease\n2018/19\n\n\n\nEach publication should have a lead statistician’s name and contact details attached to it as a requirement of the Code of Practice. You may use a team email address but the name and telephone number of a named statistician should also be provided.\nMethodology documentation is also attached at a publication level within EES - meaning one standalone piece should be written to cover all releases for the given publication within the service.\nIf you want to request a new publication, please be prepared to give the following:\n\nThe heading and subheading that you want the publication to sit under in our site (take a look at the current structure here), feel free to either use existing ones, or suggest new ones\nThe exact title of the publication\nA summary of less than 160 characters (including spaces) that describes your publication series\nIf the methodology already exists elsewhere and you have a URL for it, or if you want to create one\nTeam name\nTeam email address\nLead statistician name\nLead statistician contact number\nThe time period of the release you’re creating\nWhether they are National statistics, an ad-hoc publication, or official statistics\nList of email addresses of analysts who need access to the release and at what permission levels\n\nPublication details can be managed by publication owners or via requests to the Explore Statistics Mailbox."
},
{
- "objectID": "writing-visualising/dashboards.html#publishing-your-dashboard",
- "href": "writing-visualising/dashboards.html#publishing-your-dashboard",
- "title": "Public dashboards",
- "section": "Publishing your dashboard",
- "text": "Publishing your dashboard\n\nDfE Shiny applications are published via the DfE Analytical Services shinyapps.io account. You need to alert the statistics development team of any new dashboard publication as early in development as possible and keep us updated on expected publication date. Update the stats development team on any subsequent data or major functional updates to the dashboard publication at least a week prior to re-publishing with the update. Deploying to shinyapps requires the DfE platform codes to be entered into the repository secrets area of your app. This needs to be done by the stats development team. Authorisation of a publication should be requested from the relevant G6 or DD and the stats development team (with the former authorisation e-mail being forwarded on to the Statistics Deveopment Team).\nIf you are publishing a dashboard using already published data, then all of your code and data should be on GitHub. You may have decided to password-protect the dashboard URL, in which case, you should make the Statistics Deveopment Team aware of your publication date so that they can remove the password-protection at 9:30 on publication day, making the dashboard visible to the public.\nIf you are publishing a new dashboard for the first time that uses unpublished data, then you should have followed the guidance on using dummy data. This means that the unpublished data should not be added to GitHub until the day of publication. You should follow steps 1-9 in the below section on the day before and day of publication.\n\nBe sure to read the guidance carefully, do not commit or push unpublished data to a GitHub repo before the day of the publication of the data. If you think you may have done this by accident, contact Statistics Deveopment Team immediately with the full details of what has been uploaded to GitHub and when.\n\n\n\nUpdating data in a dashboard\n\nAs mentioned in the public dashboards section, a public dashboard should not be updated with unpublished data until the data is published. However, it is possible to clone the repo and run it locally with unpublished data for testing purposes (see our guidance on testing with unpublished data). This guidance applies to both adding real data to a dashboard that previously used dummy data, and to adding updated data to an existing dashboard.\nOn the day prior to publication:\n\nCreate a local branch which you will use to update the data. To do this open your local version of the repo, navigate to the Git window, ensure the current branch is main (see 1) as this is the code producing the current live version of the app. If you’re happy the current branch is main, then do a pull to ensure it is up-to-date (see 2). Then click the new branch button (see 3).\n\n\n\nGive your branch a descriptive name, for example, for a data update for publication on a given date, incorporate the date into the branch name (i.e. data_update_01Jan_2023). Click create.\nYou can now add your unpublished data files to the data folder in your local copy of the repository (in the file explorer). If you have already added the data using .gitignore as described in the guidance on testing with unpublished data, you can now remove the file name from the .gitignore file. You can also remove any dummy data files or old data files from the folder.\nNow open the script that reads in data for your dashboard and edit the file paths to point at the new data files.\nrun tidy_code() and run_tests_locally() (see guidance on UI tests). If changes are found when running the tests locally, make sure you look through these differences and understand them. If you are happy with the changes found in the tests (i.e. they are expected due to your updates), update the .json files. These are tested via GitHub actions every time you do a pull request, so you should always run them in advance of any pull request.\nYou can run your app to test that it is working, but do not commit or push this branch yet.\n\nOn the day of publication:\n\nAt 9:30 on publication day, commit and push the changes in your new branch to GitHub. This makes the data files publicly available via GitHub and so should not be done until 9:30 on the day of publication, however this step does not update the dashboard.\nOn the GitHub repo, you should now see a prompt to open a pull request. You should do this, following the template, ticking the boxes to show you have completed tests locally as required and provide details of the changes (i.e. that you’ve updated to include the most recent data, including dates).\nAs you have already completed tests locally (step 5), you do not have to wait for the tests to complete when you open the pull request, you can click to merge into main straight away. This should start the process of updating the data on the dashboard. You can view the progress and time taken to do this by opening the GitHub actions tab and looking at the shiny-deploy action. Once the shiny-deploy action is complete, the dashboard will have updated."
+ "objectID": "statistics-production/ees.html#admin-dashboard",
+ "href": "statistics-production/ees.html#admin-dashboard",
+ "title": "Explore Education Statistics",
+ "section": "Admin dashboard",
+ "text": "Admin dashboard\n\nUse Google Chrome or Microsoft Edge to access and use the admin part of Explore Education Statistics.\n\nWhen you enter the service you’ll see the admin dashboard, here what you can see will be dependent on your access permissions i.e. you’ll only see the publications that you have been granted access to. Within the test environment users are set up with full permissions (i.e. they can see/do everything) though in our production environment this will be restricted.\nThe Statistics Development Team will be responsible for setting up and maintaining user permissions during the beta phase. Change requests will be monitored via their EES mailbox - explore.statistics@education.gov.uk.\n\nWithin this dashboard you can view and manage existing publications, including creating and editing their releases. You can use the drop down lists to find releases by theme/topic/publication or use the draft and scheduled releases tabs to see releases that are in progress."
},
{
- "objectID": "writing-visualising/dashboards.html#whats-in-the-template",
- "href": "writing-visualising/dashboards.html#whats-in-the-template",
- "title": "Public dashboards",
- "section": "What’s in the template",
- "text": "What’s in the template\nThe template provides code for a basic interactive dashboard, with an example line chart using drop-downs for education level and geographic levels. It also includes an example accessibility statement (this is a requirement for all public dashboards and must be filled in, see the Accessibility and Accessibility testing sections earlier in this guidance), and a ‘Support and feedback’ page with suggested contact information and links.\nView the latest version of the template dashboard.\n\n\nGoogle analytics\n\nThe template has a google_analytics.html file which is set up to track all basic metrics, plus the inputs from both drop-downs. To set this up with your own app you will need a new tracking tag (the long number in the gtag()) which the statistics development team can provide you with. Please contact us in order to set this up.\nIf you are hosting your dashboard at multiple URLs (i.e. to cope with expected high traffic) then all URLs will be tracked using one tracking tag and analytics for all URLs will all appear as one in the same report.\n\n\n\nData files log\n\nThe datafiles_log.csv file is a record of all of the data files saved in the repo, and whether or not the data is published. You should log all data files here and accurately record whether the data is published, unpublished or reference data.\n\n\n\nStopping accidental data uploads\n\nIf you see any of the following errors, it is because you are trying to commit a data file to the GitHub repo that hasn’t been recorded as published:\nError: data/test_data.csv is not recorded in datafiles_log.csv.\nError: data/test_data.csv is not logged as published or reference data in datafiles_log.csv and is not found in .gitignore.\nThe template uses commit hooks and a datafiles_log.csv file. Commit hooks run automatically whenever you try to commit changes, and prevent any unpublished data being uploaded to the GitHub repo (see the public dashboards and dummy data sections) by checking the datafiles_log.csv file.You should therefore record all data in the datafiles_log and record whether it is published, unpublished or reference dara. If you try to commit a file that is not recorded or is recorded as unpublished, git will not allow the commit. If you would like to save an unpublished data file locally to test the dashboard you should use the .gitignore file to ensure git ignores this file in your commits (see the testing with unpublished data guidance)."
+ "objectID": "statistics-production/ees.html#external-user-access",
+ "href": "statistics-production/ees.html#external-user-access",
+ "title": "Explore Education Statistics",
+ "section": "External user access",
+ "text": "External user access\nAccess to EES for users outside of DfE is difficult as the authentication and security on the service relies on the DfE Azure Active Directory and associated infrastructure. It is possible to request that external users from other government organisations have access, and they can then be added as collaborating analysts on a release or as pre-release viewers.\nIf you have external users you’d like to request access for, please send the following to explore.statistics@education.gov.uk, at least two weeks in advance of requiring access. Due to the dependency on DfE’s digital security we cannot guarantee access or how long it may take.\n\nEmail addresses of users to be added\nReason for access\nLength of time access is needed"
},
{
- "objectID": "statistics-production/user-eng.html",
- "href": "statistics-production/user-eng.html",
- "title": "User engagement",
- "section": "",
- "text": "Guidance on understanding and engaging with the users of published statistics"
+ "objectID": "statistics-production/ees.html#data-and-files",
+ "href": "statistics-production/ees.html#data-and-files",
+ "title": "Explore Education Statistics",
+ "section": "Data and files",
+ "text": "Data and files\nHere you will update your data files and accompanying metadata. You need to make sure that the data has passed through our data screener checks before trying to upload it.\n\nAll data files are accessible in the downloadable files for users to explore in the same format as they are uploaded.\n\n\n\nSubject titles\n\nYou’ll need to give a ‘subject title’ to each data file you upload. The subject name should be a simple user-friendly title for the data included within the file - it is what users will see wherever the file is referenced within EES. The actual file name and data guidance can include more technical / coverage information.\nWhen adding a subject name think about the general user and how the subjects will appear in the service:\n\nYou don’t need to include the publication name in the subject name as this is always already implied within EES\nYou don’t need to list what filters are in each file in the title, users can see this in the data guidance\nYou don’t need to include the date ranges covered in each file in the title, users can see this in the data guidance\nYour subject names should be short and snappy, clearly explain what is in each file. Some good examples of this in practice are included below:\n\nEarly years provision by provider type\nExclusions and suspensions by pupil characteristics\nITT new entrants by subject and training route\n\n\nYou can make changes the subject title for your data file after it has been uploaded if needed using the ‘Edit title’ option.\n\n\n\nUploading files\n\nWhen uploading files you have a choice between uploading as separate CSV files or as a combined ZIP file.\nFor data files greater than 80mb we recommend uploading as a ZIP file.\nOnce you click to upload the file a ‘Status’ will be visible that shows the progress of the import process. It will go through the following stages:\n\nQueued\nValidating\nImporting\nComplete\n\nThis may take a little while depending on the size of your file and if there are numerous files queued for import. You cannot view the dataset or use it to create tables/charts until this status is ‘COMPLETE’.\n\nIf you believe your file is ‘stuck’ please contact explore.statistics@education.gov.uk, with details of the file and the release that you are uploading to.\n\n\n\n\nOrdering filters and indicators\n\nTo save time when creating tables and charts, and to aid users who view your data themselves via the table tool, you can now save custom orders for your filters and indicators. You can do this in the data and files page after uploading underlying data files, via the ‘reorder filters and indicators’ tab. Once there just choose which file you want to reorder and then click and drag the items until they’re in the order you want them to show in the table tool. This then becomes the default order for this filter or indicator and will apply in all charts and tables automatically.\n\n\n\n\nAncillary file uploads\n\nAny files you want to make available for users to download but aren’t intended the table tool should be added as an ancillary file upload. These files will need to meet all requirements of the new accesibility regulations before they can be published.\nExamples of ancillary files may be:\n\nInfographic pages\nSupplementary data that isn’t intended for the table tool\nAdditional technical documentation\n\nFor ensuring that spreadsheets are accessible see the guidance from .gov.wales, and this blog post that walks you through some ways that you can recreate the experience of using spreadsheets.\n\nAny data files that only consist of Planning area, and/or Institution level data should be uploaded as an ancillary file, rather than as a data file.\n\n\n\n\nPublic data guidance\n\nHere you should also create your public data guidance document, this replaces the information that would have previously been uploaded as a pdf and is designed to help users understand and use the data they download from your release.\n\nThe document will automatically update as you add new data files to your release, however you will need to add an overview of all the data included in the release and short summaries for each data file before the release can be published.\nA list of variables in each file with an associated label (taken from metadata uploads) and associated footnotes will also be displayed for each file.\n\n\nReleases cannot be published without a completed metadata document. If not filled in an error will be flagged during sign off.\n\n\n\n\nReplacing data\n\nIf you just need to change the subject title for your data file you do not need to go through the whole replacement process, just click the ‘Edit title’ option.\nHowever, if you notice a mistake in your data file you can replace it with another. When replacing a data file the platform will attempt to recreate any data blocks and footnotes that were built using the previous file.\n\nThe replacement file must contain the exact same column names and types as the original. For example, a character column named “date” must also be replaced with a character column named “date”. A numeric column named “date” will not work in the replacement.\n\n\nThe first step is to upload the new file.\n\nOne you’ve chosen and uploaded your replacement file it will need to go through the usual import process before it can check if retaining existing data blocks and footnotes will be possible.\n\nOnce the upload is finished a report will appear which highlights whether existing data blocks and footnotes can be replaced successfully. If you want to keep any data blocks and footnotes you’ve built you will need to make sure that your replacement data file still contains the information (indicators, filters, geographic_levels and time_periods) that was used to create them.\n\nIf it’s not possible for a data block or footnote to be recreated using the replacing data file a warning will appear and you’ll be prompted to either edit or delete them before completing the replacement.\n\n\nRemember to double check any data blocks or footnotes that were recreated by the platform before publishing your release."
},
{
- "objectID": "statistics-production/user-eng.html#user-hub",
- "href": "statistics-production/user-eng.html#user-hub",
- "title": "User engagement",
- "section": "USER hub",
- "text": "USER hub\nThe USER hub is now available on the GSS website, this is a central resource to help producers of statistics develop the knowledge, skills and techniques needed to engage effectively with their audiences."
+ "objectID": "statistics-production/ees.html#footnotes",
+ "href": "statistics-production/ees.html#footnotes",
+ "title": "Explore Education Statistics",
+ "section": "Footnotes",
+ "text": "Footnotes\nWe generally advise against using footnotes, unless they’re absolutely necessary (think about how many people will actually be reading them!). Consider where you could signpost to the methodology /data guidance / commentary instead. However, if you do want to add footnotes to your data, this should be done using the EES footnote function.\nFootnotes are added via footnotes tab - rather than writing multiple tables and assigning individual footnotes, you write footnotes and assign them to certain indicators and filters so they appear when users select them in the table builder. For example in the below, the footnote “This is a footnote” is assigned to “Headcount” indicator for all options within the “School type” filter.\n\nIf you’d rather, you can assign a footnote to the whole data file by ticking this box.\n\nYou can assign the same footnote across multiple data files.\n\nWe recommend that you only add footnotes once you are certain the data file is final. If you have to delete the data file, all the assigned footnotes will be deleted alongside it."
},
{
- "objectID": "statistics-production/user-eng.html#improving-your-own-engagmenent",
- "href": "statistics-production/user-eng.html#improving-your-own-engagmenent",
- "title": "User engagement",
- "section": "Improving your own engagmenent",
- "text": "Improving your own engagmenent\nDownload this top tips document produced for the GSS, as a starting point for understanding how you can improve your user engagement.\nThrough our work engaging users on EES we have built up a list of interested users who have stated a willingness to be contacted about opportunities to be involved in user testing of our statistics on the platform, if you have any suggestions for things for us to test on EES with these users, please contact explore.statistics@education.gov.uk.\nMore advice and guidance will follow in this space. In the meantime, if you’re interested in user engagement please contact Heather Brown."
+ "objectID": "statistics-production/ees.html#data-blocks-tables-and-charts",
+ "href": "statistics-production/ees.html#data-blocks-tables-and-charts",
+ "title": "Explore Education Statistics",
+ "section": "Data blocks (tables and charts)",
+ "text": "Data blocks (tables and charts)\nA data block is a smaller cut of data from your original file that you can embed into your publication as a presentation table, build charts from, or link users directly to.\n\n\nTables\n\nHere you can create data blocks, using the admin table tool to create and save your summary tables. Remember to use the reorder table headers tool to restructure your table however you want it before you save.\nOnce you’re happy with your table, give it a title and some source information before clicking save. The source should be the source of the data used to create the data file that was then used for the data block table, for example ‘School Census’.\nThe manage data blocks page will list all the data blocks you have created, highlighting which have charts, are used in content and are saved as highlight tables.\n\n\n\nFeatured tables\n\nYou can also choose to highlight a data block table as a ‘featured table’ which means it will show in a list of featured tables within the table tool. This is designed to help users get to tables of interest more quickly (without having to create tables themselves).\nThere is an option to choose if a table is a ‘featured table’ when saving each data block, here you can name the table and add a description giving the table coverage:\n\nEach featured table will then be listed to the user within the table tool. Featured tables do not have to be embedded within your release content to be included in this list.\n\n\n\n\nFast track links\n\nAny data block tables that are created and saved within the admin app will also be assigned a ‘fast-track link’, this url can then be used throughout your release as a way to direct users to specific tables (within the table tool) more quickly so they can interact and explore the data further. It will appear at the top of the page like this:\n\nWhen your release is published, any embedded data block tables within the release will have an ‘explore data button’ beneath them which will utilise these fast track links to quickly direct users to the table within the table tool so they can explore the data further. You can also use fast-track links as a hyperlink within release commentary (without having to embed the data block).\nIn your fast-track titles, you don’t want to overload information, but still want to direct the user to the right place. Remember they can go back to the table tool through your featured tables to change filters and indicators as needed.\nFast-track titles should explain:\n\nWhat the table is showing in the simplest terms\nWho/Where the data covers (e.g. characteristic groups and geography levels)\nWhen the data in the table is reported for\n\nHere are a few examples of good fast-track titles in EES:\n\nNumber of Schools and Pupils, by School Type, 2015/16 to 2020/21\nAbsence Rates by School Type, 2016/17 to 2020/21\nFree School Meals, by Region, 2015/16 to 2020/21\n\n\nWhat is the difference between a fast-track link and a permalink?\nEES also offers ‘permalinks’ for any table created in the table tool which allows a user to save a link to a permanent, static, version of a table they have created. Analysts can make use of these permalinks when answering queries or in PQ and FOI responses.\nFast-track links are similar to permalinks however instead of linking to a static version of a table they link to an ‘active’ version of the table within the table tool - meaning users can interact and change what’s shown in the table from within the table tool if they choose to.\n\n\n\n\n\nCharts\n\nAfter building and saving a data block table you will see a ‘Chart’ tab appears. This tab will take you to the EES chart builder, where you can choose to add a chart to your data block.\nThe first step to creating a chart is choosing the chart type, currently the EES chart builder can build line charts, horizontal/vertical bar charts (including stacked and clustered) and maps.\nAfter choosing your chart type you then need to work through the following stages to build your chart. In each stage you’ll be shown a live preview as you make changes.\n\n\nChart configuration\n\nWithin the ‘chart configuration’ tab you can add a title, alt text, move the legend and change chart dimensions.\n\nMake sure to review your chart dimensions before you publish. Users should be able to read the labels on the axes and see the legend without having to scroll.\n\n\nNote, within the vertical and horizontal bar chart types you can also create stacked bar charts by clicking the ‘Stacked bars’ option within the chart configuration tab\n\n\n\nData sets\n\nHere is where you add data to the chart. You can add each series one at a time or all together.\n\n\n\n\nLegend\n\nYou can edit the chart legend, and styling of your series via the Legend tab.\n\n\nTo select specify custom colours outside of the defaults, you can double click on the colour codes at the bottom of the colour picked until you get to the type of code you’re wanting to input (e.g. hex code) and then enter the code manually.\n\n\n\n\nX axis (major axis)\n\nHere is where you configure the x-axis: You can alter gridlines, labels, sort, limit and add reference lines.\n\n\n\n\n\n\n\n\nOption\nWhat it does\n\n\n\n\nSize of axis\nChange the width of the space given to axis tick labels\n\n\nShow grid lines\nTurn grid lines on and off\n\n\nShow axis\nTurn the axis on and off, you can also add a unit to the axis tick labels\n\n\nSorting\nChange how the data within the chart is sorted\n\n\nTick display type\nAlter how often axis tick labels are shown, labels will automatically skip values where there are too many to show without overlapping\n\n\nAxis range\nAlter the range of data shown in the chart\n\n\nLabel\nAdd an axis label, you can also choose the width for the space given to it\n\n\nReference lines\nAdd/remove reference lines to the chart\n\n\n\n\n\n\nY axis (minor axis)\n\nThen follow a similar process for the y axis configuration, play around until the chart looks how you want it to.\n\n\n\n\n\n\n\n\nOption\nWhat it does\n\n\n\n\nSize of axis\nChange the width of the space given to axis tick labels\n\n\nGroup data by\nChange how the data within the chart is grouped\n\n\nShow grid lines\nTurn grid lines on and off\n\n\nShow axis\nTurn the axis on and off, you can also add a unit to the axis tick labels\n\n\nSorting\nChange how the data within the chart is sorted\n\n\nTick display type\nAlter how often axis tick labels are shown, labels will automatically skip values where there are too many to show without overlapping\n\n\nAxis range\nAlter the range of data shown in the chart\n\n\nLabel\nAdd an axis label, you can also choose the width for the space given to it\n\n\nReference lines\nAdd/remove reference lines to the chart\n\n\n\n\n\n\nChanging chart type\n\nIf you create your chart and then change your mind as to what chart type would be best you can just click to change it and it will try to save all the options that you had applied previously.\n\n\nRemember to save your chart when you’re done.\n\n\n\n\nMaps\n\nYou can create maps too, currently this is possible for regional, LA and LAD data.\nYou can change the boundaries you are plotting onto via the “chart configuration” tab, the latest boundary file will automatically be selected, but if you are visualising historic data, you may want an older boundary file. Please contact us if the boundary you want to plot is unavailable.\n\nThen to create a map, add the cuts of data you want to display in the “data sets” tab of the chart builder.\n\nYou can change the colour scale of the chart in the “legend” tab.\n\n\n\n\nOther chart types\n\nYou should only use complex charts where there is a clear user need, as simple charts are the easiest for users to understand. If you have a an infographic or a chart that is too complex to build within our chart builder you can use the ‘Choose an infographic as alternative’ option to upload an image to your data block or use the Add embed block feature to embed an R-Shiny based plot (see the section on embedded blocks for further information).\nInfographic alternatives should be .svg format and you can use the sizing options within the data block editor to size your image appropriately.\n\nAccessibility matters for infographics too, consider the following if you do need to upload an image -\n\nKeep them simple\nUse colours that are available in EES - see our visualisation guidance for more details\nDon’t put borders around images\nRead the GSS guidance on the use of colour in visualisations and follow the steps provided to check your visualisations\nTry to avoid adding text to your images other than axis labels and limited annotations. Titles and headings can be added in the “chart configuration” tab instead\n\n\nYou can use R to create infographics and the following code gives an example of how to create a basic line chart or stacked bar chart using the appropriate GSS sequential colour palette.\n# Load the necessary libraries\nlibrary(ggplot2)\nlibrary(dplyr)\n\n# This is the GSS standard categorical colour palette\ngss_categorical_palette <- data.frame(\n names = c('Dark Blue', 'Turquoise', 'Dark pink', 'Orange', 'Dark grey', 'Light purple'),\n hex = c(\"#12436D\", '#28A197', \"#801650\", \"#F46A25\", \"#3D3D3D\", \"#A285D1\"),\n id = c(1,2,3,4,5,6))\n\n# Set up a dummy data-set.\n# Note the line factor(...,levels=...) allows you to order your filter values in\n# the final plot based on the ordering entered into the levels keyword. If left,\n# it'll default to alphabetical.\ndata <- data.frame(\n time_period=c(\"2018/19\",\"2018/19\",\"2018/19\",\"2018/19\",\"2018/19\",\"2018/19\",\n \"2019/20\",\"2019/20\",\"2019/20\",\"2019/20\",\"2019/20\",\"2019/20\",\n \"2020/21\",\"2020/21\",\"2020/21\",\"2020/21\",\"2020/21\",\"2020/21\"),\n Filter=c(gss_categorical_palette$names,\n gss_categorical_palette$names,\n gss_categorical_palette$names),\n indicator1=sample(4:16,18,replace=TRUE) +\n c(gss_categorical_palette$id,gss_categorical_palette$id,gss_categorical_palette$id)*3\n ) %>%\n mutate(\n time_period=as.factor(time_period),\n Filter=factor(Filter,levels=gss_categorical_palette$names)\n )\n\n# Create a line chart\nggplot(data, aes(x=time_period, y=indicator1, group=Filter, colour=Filter )) +\n geom_line(size=1.2) +\n scale_color_manual(values=gss_categorical_palette$hex) +\n theme_classic() +\n theme(\n legend.position = \"bottom\",\n text = element_text(size = 14, family = \"Arial\"),\n strip.text.x = element_text(size = 20)\n ) +\n ylab(\"Indicator 1\") +\n xlab(\"Time period\")\n\n# Create a stacked bar chart\nggplot(data, aes(x=time_period, y=indicator1, fill=Filter )) +\n geom_bar(stat = \"identity\") +\n scale_fill_manual(values=gss_categorical_palette$hex) +\n theme_classic() +\n theme(\n legend.position = \"bottom\",\n text = element_text(size = 14, family = \"Arial\"),\n strip.text.x = element_text(size = 20)\n ) +\n ylab(\"Indicator 1\") +\n xlab(\"Time period\")\nThe above code should produce something along the lines of the following plots:"
},
{
- "objectID": "statistics-production/user-eng.html#case-studies",
- "href": "statistics-production/user-eng.html#case-studies",
- "title": "User engagement",
- "section": "Case studies",
- "text": "Case studies\n\nExplore Education Statistics service\n\nExplore Education Statistics was born from a discovery project, where the emphasis of the discovery was to understand who the users of the Department’s official statistics are, what they need, and where the DfE are currently meeting, exceeding or failing to meet those needs. We spoke to over 90 users directly, via 1 to 1 sessions or workshops and had over 130 survey responses. We used this research to create user personas and user stories which we then assessed ourselves against to see how well we were meeting those needs - the result of this was a recommendation that we needed to do something better. Our discovery report is published on gov.uk.\nFollowing the discovery we moved through to the alpha phase, where we built prototypes and carried out extensive user testing to firm up the functionality that might be required - focussing on different user journeys and the user experience at each step. We then passed a GDS service assessment where our approach was tested against the GDS service standard before moving into the private beta phase where we started building the service for real. We user-tested every bit of the journey thoroughly and used feedback from users to iterate on the functionality we were making available. User testing was primarily through 1 on 1 sessions where users would be set activities to work through and we would watch how the user clicked through each part of the relevant journey and to ask what they might expect to see at any different steps of the process - both when creating/publishing statistics releases and using releases after publishing. We also ran workshops and carried out user surveys to collect as much evidence as possible, for more information on this take a look at the write up of our private beta phase on Hive IT’s website.\n\nIn March 2020 we wrapped up the build of the service and were given the green light via a GDS service assessment to move into our public beta phase. That’s when we started using the service for real and teams started publishing statistics via EES only. Ongoing user feedback has been really important throughout public beta as it has helped us move the service from a minimal viable product to one that works in the best way possible for everyone. We also use this engagement to help us understand how well we are meeting our KPIs, which we review regularly to see how well we are meeting the original aims of the service thinking back to our original discovery and what users told us they needed. There are a number of ongoing feedback routes open to users during the public beta phase of the service:\n\nBeta banner (service focus) Users can submit feedback through the beta banner from any EES page, and responses are managed by the statistics development team. Feedback so far has mostly been aimed at the service functionality more generally but we do regularly get publication specific feedback which we share with the relevant teams.\nFormal user testing (service focus) Our EES team will speak to users via user testing sessions to help inform future iterations to functionality.\nIn-release feedback requests (publication focus) Some teams have been including specific calls for feedback within their release pages e.g. - https://explore-education-statistics.service.gov.uk/find-statistics/apprenticeships-and-traineeships\nGoogle analytics We collect user analytics to help us understand what users are doing with our statistics. This information is shared with production teams via our EES analytics app.\n\nFor more information, get in touch with the team at explore.statistics@education.gov.uk.\n\n\n\nSchool places local authority scorecard\n\nIn 2021 the annual ‘Local Authority School Places Scorecard’ publication was cancelled, due to Covid- 19, allowing time for user-engagement and development of the scorecard. This is a dashboard publication allowing users to select a specific local authority in England, to view it’s progress in providing quality school places. Firstly, internal feedback was sought and received, and generally users found the Scorecard a useful tool. Longer-term additions were identified, however short-term development was also needed to improve the accessibility of the Scorecard so this was the first developmental focus. The Scorecard has been previously published as an excel-based dashboard, however this method of publication was no longer viable due to new rules and regulations for official statistics publications. A new platform was therefore needed and R shiny was suggested as an accessible platform. A new version of a previously published scorecard was produced in R shiny. This was then published externally to gather feedback from users to check they could obtain the required information. The scorecard contained a link to a short survey and over 90 internal and external users responded (after email invitations were sent allowing a month to complete). Most users found the R shiny version easier to use compared to the excel based dashboard which was positive. Suggestions for improvement were gathered and quite a few users mentioned they missed the one-page format of the excel scorecard (the R shiny version has tabs to get to some visualisations) and found some of the newer charts slightly confusing. Therefore solutions were proposed e.g. a downloadable one page PDF summary and alternative charts/clearer guidance. Development work is still ongoing with the new R shiny scorecard due for publication in June 2022. When published this should include improvements to address the most frequent suggestions mentioned in the survey, to ensure users can gather the information they need from the scorecard. For any further information contact Natalie Paterson in the Pupil Place Planning Data team.\n\n\n\nCIN-CLA outcomes statistical release\n\nIn 2020 we created a new statistical release which incorporated changes to the annual Outcomes for children looked after by LAs statistical release and outcomes within the Characteristics of children in need statistical release, proposed by the department in September 2020. Improving these statistics was a commitment made at the end of the CIN review. As part of the review of the National Statistics, an ad-hoc release was published (Outcomes of children in need including looked after children) and we asked users to submit feedback on the proposals through the running of a consultation via Gov.UK to which people could respond (Children in need and looked after children statistics: proposed changes). We involved policy colleagues (including policy analysts) in the proposed changes ahead of publishing. We engaged with colleagues across the department to identify known users and user groups and notified them of the request for feedback on the proposals We also followed up with a reminder as we got closer to the deadline. As part of this process, we had a meeting with the main stakeholders (the board of the National Association of Virtual School Heads). These key were very keen to respond to the proposals but were having difficulty with resource and time to provide a comprehensive response, so we agreed to have a meeting to talk through everything and we took notes. Reaching out to users and actively engaging with them produced a great result - we received a wide range of feedback from a range of sources including previously unknown users. We received external and internal feedback. We published the results of the feedback on the proposals on gov.uk, including a summary of the feedback received and what we were able to incorporate into the new publication. This was not an official Government consultation and we had to negotiate with gov.uk on how to publish the documents. We were advised that going forward, it would be preferable for user engagement of this type to be published alongside the statistical releases as part of the new EES platform. For any further information contact Bree Waine."
+ "objectID": "statistics-production/ees.html#content",
+ "href": "statistics-production/ees.html#content",
+ "title": "Explore Education Statistics",
+ "section": "Content",
+ "text": "Content\nIn the content tab you can now start creating your release, embedding the data blocks you’ve created as you go.\n\nYou can use the page view toggles, that float in the bottom left of the page, to jump between edit and preview mode for the release and to view a preview of the table tool.\n\nAdd any headline and/or key stats and figures for your release in the headline facts and figures section.\nYou can then create accordion sections to start adding your main release commentary. These sections are made up of text blocks and data blocks which can be reordered as needed.\n\nHeadline facts and figures\nUse the ‘Add secondary stats’ button to add a data block to your headline stats section.\nUse the ‘Key stats’ options to add key statistic tiles to your release. For each tile you first have to have created a data block that contains only one number.\n\nAfter embedding a key stat tile you can then edit it to add trend information and a description of what the indicator is.\n\n\n\n\n\n\n\nElement\nContent\n\n\n\n\nIndicator name\nAutomatically generated from your data\n\n\nLatest value\nAutomatically generated from your data\n\n\nTrend\nA short one-sentence description of the trend; try to avoid only stating the change from the previous year and talk about the longer-term trend where appropriate\n\n\nGuidance title\nE.g. ‘What is NEET?’ or ‘What are permanent exclusions?’\n\n\nGuidance text\nA simple description in plain English of what the indicator is\n\n\n\n\n\n\nDrafting text\n\n\nAbout these statistics\n\nWe recommend that the first accordion section in each release should be ‘About these statistics’. Do not assume that users will read it, the nature of interactive pages means that the reader may start at any accordion section so remember to include essential information in the section to which it refers.\nAvoid filling this section with:\n\nCrucial caveats or information necessary for accurate interpretation of the statistics, these should be included within the main commentary next to the point they relate to\nTechnical information that is not relevant to the interpretation of the statistics should be saved for the methodology\nDefinitions should be included within the main commentary when they are first mentioned\nLists or descriptions of what is in the commentary as these should be clear from the contents and accordion headings\n\nTry to focus the ‘About these statistics’ section on:\n\nSay why the data are collected and what they are used for or could be used for, including relevant policies and targets\nThe different geographical levels for which data is published\nNon-essential but important information about the statistics, such as:\n\nThe data collection window, mode of data collection and response rates in terms of whether these are representative\nClarification of the population coverage\nChanges in data collection or analysis methods from previous releases\n\nLinks to relevant related statistics including the cause of any differences\nFurther useful detail on specific measures used in the release\nNon-urgent notices to users such as minor error corrections and delays or changes to the next publication. If there is something important which you want all users to know about then give it its own accordion section.\n\n\n\n\n\nFootnotes\n\n\nDo not use footnotes in the text of your content. They’re designed for reference in print, not web pages. If the information in the footnotes is important, include it in the body text. If it’s not, leave it out.\n\n\n\nTables in content\n\nAny data tables should be included as data blocks, however you can also embed static html tables within a text box. These should only be used to present textual tables or for any small presentations of data that are not possible to do in a data block at the moment.\n\n\nYou can create static html tables for presenting information that isn’t embedded in a data block. However, remember that all of the data included or referred to in your content should be available (or createable) from the downloadable open data files.\n\n\n\nFootnotes for tables in content\n\nIf you are including a table in text that needs footnotes, it’s generally advised to include this in the commentary surrounding the table. However, if you think a footnote is still necessary, then we advise writing out the word ‘note’, with the number of the note you need to refer to, and put it in square brackets, for example: ‘Number of people in employment [note 1]’. For more guidance on footnotes outside of EES, see the Analysis Function guidance on symbols, footnotes and codes.\n\n\n\n\nReviewing content\n\nWhile a release is in draft mode, comments can be added to text to help teams collaborate. Simply highlight the text you want to comment on and click on the speech bubble in the editing bar to add a comment.\n\nComments can be edited at a later date, and can also be marked as resolved so that you can see which comments have been addressed and which are still outstanding.\nWhen someone is editing a text box, it will now be instantly frozen for all other users preventing two users from editing the same block of text at the same time. You will be able to see the name of the user who is making edits, and will see the edits coming through every few seconds as their changes autosave."
},
{
- "objectID": "statistics-production/user-eng.html#what-is-user-engagement",
- "href": "statistics-production/user-eng.html#what-is-user-engagement",
- "title": "User engagement",
- "section": "What is user engagement?",
- "text": "What is user engagement?\n\n\n\n\nThe ONS have also published ten top tips to help improve user engagement."
+ "objectID": "statistics-production/ees.html#embedding-r-shiny-blocks",
+ "href": "statistics-production/ees.html#embedding-r-shiny-blocks",
+ "title": "Explore Education Statistics",
+ "section": "Embedding R-Shiny blocks",
+ "text": "Embedding R-Shiny blocks\nIf you need to include a type of chart that isn’t possible using the standard EES chart options, for example an interactive chart with filters, you can embed a block to display a custom R-Shiny produced chart.\nTo embed a shiny chart, you’ll first need to create an R-Shiny application containing the chart using the DfE tiny-shiny template and get it hosted on our DfE GitHub and ShinyApps accounts. More details on both of these are available in the tiny-shiny section of this guidance.\nOnce you’ve got the R-Shiny app set up and hosted, you can embed it using the Add embed block button in the content area. This will give you the option to enter a URL, where you can enter the URL of the shiny app. Valid URLs to use in the embed block dialogue box are limited to only those on the DfE ShinyApps server (and the internal rsconnect when needed for draft publications).\n\n\n\nThe EES interface for embedding a Shiny chart"
},
{
- "objectID": "statistics-production/user-eng.html#user-engagement-successes",
- "href": "statistics-production/user-eng.html#user-engagement-successes",
- "title": "User engagement",
- "section": "User engagement successes",
- "text": "User engagement successes"
+ "objectID": "statistics-production/ees.html#glossary",
+ "href": "statistics-production/ees.html#glossary",
+ "title": "Explore Education Statistics",
+ "section": "Glossary",
+ "text": "Glossary\nOur glossary on EES is a growing page that helps us to standardise how refer to key terms and phrases across all of Official statistics - https://explore-education-statistics.service.gov.uk/glossary.\n\n\nAdding new definitions\n\nContact us at explore.statistics@education.gov.uk, with the title and definition and we can add this for you. It’s worth having a check on the glossary for similar or related terms, and whether you should be looking to harmonise with other teams.\n\n\n\nLinking to definitions\n\nYou can link to any term in the glossary by appending the glossary url with #name-of-definition, replacing any spaces with hyphens.\nFor example, to link to the definition for ‘Respite care’, you would use the following link:\nhttps://explore-education-statistics.service.gov.uk/glossary#respite-care\nYou can test this works by typing the url into your browser, it should take you to that specific definition on the glossary page. If you’re unsure at all, or have special characters in the title of your glossary entry, please ask us for support on getting the right link.\n\n\n\nLinking from your release\n\nWhen writing your release content, highlight a word or phrase and click the link icon in the text editor bar.\n\nThen, paste your glossary url into the box that appears.\n\nThat’s it, the system will automatically recognise that the link is for the glossary and will do the rest. You can then change to the preview mode to see how this would then appear to public users, and test that the box appears with the definition."
},
{
- "objectID": "statistics-production/user-eng.html#consultations",
- "href": "statistics-production/user-eng.html#consultations",
- "title": "User engagement",
- "section": "Consultations",
- "text": "Consultations\nIf you are considering changes to your publication and are contemplating a formal consultation do get in touch with HOP office early on to discuss.\nTo read more about consultations for background information please see the following two links:\n\nCode of Practice on Consultations\nConsultation principles: guidance - WWW.GOV.UK\n\nWith the first of the above links, please bear in mind this was written back in 2008 so certain aspects may have been updated but the background and principles do remain the same.\nSome examples of previous consultations are included below:\n\nOutcome and experience data - Office for Students\nConsultation on Data Futures and data collection - Office for Students\nSurveys on childcare and early years in England - GOV.UK (www.gov.uk)\nProposed cessation of ‘Income related benefits: estimates of take-up’ statistics - GOV.UK (www.gov.uk)\nProposals for a new statistical series to count unemployed claimants - GOV.UK (www.gov.uk)"
+ "objectID": "statistics-production/ees.html#sign-off",
+ "href": "statistics-production/ees.html#sign-off",
+ "title": "Explore Education Statistics",
+ "section": "Sign off",
+ "text": "Sign off\nOnce you’re happy with your release you need to go to the sign off page and change it’s status in order to move it through the release pipeline.\nThere are three statuses:\n\nIn draft (where the production team work on drafting the release)\nReady for higher review (where the senior statistician checks over the release before approving)\nApproved for publication (after approval has been given, releases in this status will be published on their scheduled date)\n\n\nOnly users with approver permissions (usually G6 or above) can sign off the release for publication.\n\n\n\nErrors and warnings\n\nThere’s quite a few things to remember to do as you build your release so to help ensure you haven’t missed anything a release checklist is also available via the sign off page\nThe checklist can be accessed by clicking to edit release status.\nRemember to check over it before you submit your release for approval as a release that has outstanding warnings on it will not be able to be published.\n\n\n\n\nHow to approve a release\n\nReleases are approved via the sign off tab. The release date is also set during this stage of the process. All releases scheduled for a specific date will be published at 9:30am on that day.\n\nThe approver has the ability to approve the release to be published on a specific date or to publish as soon as possible. Publishing as soon as possible is useful for publishing amendments to existing releases.\n\nThis page also gives you the expected release url which may be useful to know for other things, for example, sending to the web team to add to your gov.uk announcement page.\n\n\n\n\nNext release expected\n\nYou also have the option to add a date for when the next release is expected. This will appear at the top of a release page and give users an idea of when to expect the next release in the publication series. You can provide the planned month and year of the next release within this publication.\nIf this is added but then needs to be changed or removed later in the year it is easy to do so, by creating an amendment and setting a new next release expected date on sign-off."
},
{
- "objectID": "statistics-production/user-eng.html#surveys",
- "href": "statistics-production/user-eng.html#surveys",
- "title": "User engagement",
- "section": "Surveys",
- "text": "Surveys\nWe routinely publish statistical releases, but who is using these and why? It’s important to understand your user types and need, so we (public and department) get best value from what we do. Using these insights can help us to continually improve our products and services. Having a survey with your releases is a great way to glean this feedback. Examples of statistical publication surveys, as well as ONS top tips on user surveys are below/attached. It would be great to include your examples too, as well as capture your lessons learned and tips, so do send these to us. If you do want further advice, please contact Heather Brown or Ian Hewson.\nUser survey and questionnaire design top tips (word document)\nResults from survey of statistics users (word document)\n\n\nExamples\n\nHigher Education Institutions Enrolments and Qualifications statistical bulletins - readership survey September 2019 (economy-ni.gov.uk)\nFurther Education & skills/Apprenticeships and traineeships release feedback survey - IH (Edit) Microsoft Forms (office.com)\nHigher Education Statistics - Department for Education - Citizen Space\nStatistics-and-data_user-feedback-survey_Land-and-Property-Data-Team_RoS.._.pdf"
+ "objectID": "statistics-production/ees.html#pre-release-access",
+ "href": "statistics-production/ees.html#pre-release-access",
+ "title": "Explore Education Statistics",
+ "section": "Pre-release access",
+ "text": "Pre-release access\nPrior to each release going live the production team are also able to grant pre-release access to a named group of users 24 hours before it goes live. These users do not require full access to the whole admin service. They will be able to see preview versions of any releases they have been granted access to.\nThis preview is only accessible for the 24 hours before the publication date, the emails may go out to users before then. We expect teams will continue to send an email at 09:30 on pre-release day, including any additional briefing and the link to the pre-release area.\n\nJobshare emails should not be sent invites for pre-release. The personal emails for the specific individuals should be used instead, as jobshare emails do not have active accounts with DfE to access EES.\n\n\nPre-release access via Explore Education Statistics is currently limited to DFE users only, if you need to share your release with external users you will need to do so outside of the system.\n\n\n\n\n\nEES PRA one pager\n\n\n\n\n\n\n\n\n\n\n\nInviting users for pre-release access and building the public pre-release list can be found within the ‘Pre-release access’ tab on the dashboard.\nOnce the release has been marked as approved, go to the ‘Pre-release access’ tab and add the relevant email addresses to grant pre-release access. All invited users will receive an email to say that they have been given pre-release access and will get a url where the preview release will be available.\n\nYou should also create the public facing pre-release access page by clicking the ‘Public access list’ tab.\n\nAfter creating your pre-release access list a text editor will appear where you can list the roles that have been given early access to the release. This list will then appear in the list of file downloads at the top of each release."
+ },
+ {
+ "objectID": "statistics-production/ees.html#release-notes",
+ "href": "statistics-production/ees.html#release-notes",
+ "title": "Explore Education Statistics",
+ "section": "Release notes",
+ "text": "Release notes\nWhen publishing a new amendment you should add a ‘release note’ to your release so users can be informed of what has changed:"
+ },
+ {
+ "objectID": "statistics-production/ees.html#methodology-formatting",
+ "href": "statistics-production/ees.html#methodology-formatting",
+ "title": "Explore Education Statistics",
+ "section": "Methodology formatting",
+ "text": "Methodology formatting\nSubheadings make it easier for your users to navigate through your methodology. When editing text blocks in your methodology, you can do this by highlighting your subheading and selecting “Heading 3” in place of “Paragraph”:\n\nYou can add further subheadings underneath this but try to limit the number of subheadings per accordion to 10 at the most. Too many subheadings will make it tricky for users to find what they are looking for."
+ },
+ {
+ "objectID": "statistics-production/ees.html#methodology-images",
+ "href": "statistics-production/ees.html#methodology-images",
+ "title": "Explore Education Statistics",
+ "section": "Methodology images",
+ "text": "Methodology images\nYou can add images to methodology pages via the content editor.\n\nWhen uploading an image you will need to add alt text via by clicking the eye symbol, you may also choose to add an image caption.\n\n\nAccessibility matters for uploaded images too, consider the following if you do need to upload an image -\n\nKeep them simple\nUse GDS colours\nDon’t put borders around images\nEnsure there is a high enough colour contrast ratio between elements. You can use a colour contrast checker to check the colours you’re using\nTry to avoid adding text to your images other than axis labels and limited annotations."
+ },
+ {
+ "objectID": "statistics-production/ees.html#publication-management",
+ "href": "statistics-production/ees.html#publication-management",
+ "title": "Explore Education Statistics",
+ "section": "Publication management",
+ "text": "Publication management\nUse this page to change the details for your publication, such as the contact information or to add links to legacy releases on gov.uk.\nAny updates here will affect all releases in the series and will be published immediately.\n\nIt is possible to change the publication title via this page, however this should be used rarely and will not affect the publications url if one or more published releases exist."
+ },
+ {
+ "objectID": "statistics-production/ees.html#create-new-release",
+ "href": "statistics-production/ees.html#create-new-release",
+ "title": "Explore Education Statistics",
+ "section": "Create new release",
+ "text": "Create new release\nAfter finding the publication you want to create a release for, just press the button to create a new release.\nCurrently the following types of release can be created in EES:\n\nNational Statistics (includes OSR tick mark logo at the top of the release page)\nOfficial Statistics\nAdhoc Statistics\n\nWhen creating a release you will be asked to fill in some release summary fields.\n\nThe time period for the release should reflect the time period of the data that this latest release adds to the time series."
+ },
+ {
+ "objectID": "statistics-production/ees.html#managing-legacy-releases",
+ "href": "statistics-production/ees.html#managing-legacy-releases",
+ "title": "Explore Education Statistics",
+ "section": "Managing legacy releases",
+ "text": "Managing legacy releases\nPast publications on other services can be added to the previous releases links that appear on the top right of a release page by using the ‘Manage legacy releases’ section. In here you can add links to previous releases and choose the order in which they appear.\nThis section can be found by publication owners in the publication management section."
+ },
+ {
+ "objectID": "statistics-production/user-analytics.html",
+ "href": "statistics-production/user-analytics.html",
+ "title": "EES analytics",
+ "section": "",
+ "text": "Guidance on understanding how users are interacting with your statistics published via EES\n\n\nWe track analytics of users on Explore Education Statistics using Google Analytics as well as other tools. These analytics can be found on our EES Analytics dashboard. This tells you a range of things like:\n\nHow many times each accordion has been clicked\nHow long users spend on your page\nHow many times each table has been used in the table builder or downloaded\n\n\n\nUser-led content improvements\n\nThe below table gives some pointers on how you can use these stats to improve your release for users:\n\n\n\n\n\n\n\nMeasure\nGuidance\n\n\n\n\nSearch terms\nFor popular search terms, add additional commentary or signposting using subheadings and accordions for to support your user’s interests. If you refer to a popular search term by another name, consider changing it so users know what you’re talking about.\n\n\nAccordion openings\nConsider removing accordions with low numbers of clicks from your release. You could move them further up the page or group them into existing accordions if this commentary is essential to your release.\n\n\nDownloads\nFor files with low numbers of downloads, consider combining information into other files, or consider removing from your publication entirely if users are no longer interested in this data.\n\n\nTables built\nFor tables with low numbers of interactions in the table tool, create featured tables to demonstrate the useful information they hold.\n\n\nPermalinks\nSave popular permalinks as featured tables in future, so users can easily get to the data they want.\n\n\nTraffic sources\nConsider where your users are being routed from and what is being written about your stats - clearly highlight key stories in the Headlines section and make use of the blue key stat tiles to make them easy to digest.\n\n\n\n\n\n\nGuidance for analytics\n\nPerformance analysis: GDS starter guide to using Google Analytics\nPerformance analysis: GDS Advanced Analytics Reporting Techniques\n\n\n\n\n\n Back to top"
},
{
"objectID": "statistics-production/ud.html",
@@ -212,219 +261,135 @@
{
"objectID": "statistics-production/ud.html#file-size",
"href": "statistics-production/ud.html#file-size",
- "title": "Open Data Standards",
- "section": "File size",
- "text": "File size\nThere are no character or size limits in a csv file and there is no size limit for EES, though the larger a file is, the longer it will take to upload and process. Also remember that the files you upload are the files that users will download, consider the software they may access to (e.g. Excel) and whether the size of your files are compatible with this.\nExcel has a cell character limit of 32,760 and a row limit of 1,048,576. It is best to avoid exceeding these as some end users may struggle to open the file. One good way to cut the file down is to split after a certain number of years, or to separate out different geographic levels into separate files, providing school level data as a separate file for example. With the data all being in a tidy format these are then easy enough for secondary analysts to stitch back together if needed.\nA rough guide to file size would be:\n\nAnything under 10mb is relatively small\n10mb to 100mb is a fairly common file size that most teams have\n100mb to 500mb is a large file and will struggle to upload if not compressed to a zip folder.\n500mb and over are very large, and sometimes may struggle to compress small enough to upload.\n4gb or more in size is larger than any we have seen before and will likely need testing in the platform first.\n\nContact us if you have any issues, or files that might be over 1gb."
- },
- {
- "objectID": "statistics-production/ud.html#data-symbols",
- "href": "statistics-production/ud.html#data-symbols",
- "title": "Open Data Standards",
- "section": "Data symbols",
- "text": "Data symbols\nIn line with the GSS guidance on symbols, special values should be replaced with symbols in the following situations:\n\n\n\n\n\n\n\n\n\nSymbol\nUsage\nExample\nObsolete equivalents\n\n\n\n\nz\nWhen an observation is not applicable\nNo data for at gender level for boys at an all-girls school\n\n\n\nx\nWhen data is unavailable for other reasons\nData for an indicator is not collected in a certain region\n:\n\n\nc\nConfidential data\nData has been suppressed\n\n\n\nlow\nRounds to 0, but is not 0\nRounding to the nearest thousand, 499 would otherwise show as 0. Only use 0 for true 0’s\n~\n\n\nu\nWhen an observation is of low reliability\nData for a local authority is identified as missing returns so is removed from regional and national totals\n\n\n\n\nIf you have any other conventions you’ve used in previous publications, or a scenario that isn’t covered above, check the GSS guidance (ignoring the part around separate columns for symbols), and contact us."
- },
- {
- "objectID": "statistics-production/ud.html#mandatory-ees-metadata-columns",
- "href": "statistics-production/ud.html#mandatory-ees-metadata-columns",
- "title": "Open Data Standards",
- "section": "Mandatory EES metadata columns",
- "text": "Mandatory EES metadata columns\n\n\n\n\n\n\n\ncolumn\ndetails\n\n\n\n\ncol_name\nThis must exactly match the name of the corresponding column in the dataset.\n\n\ncol_type\nThis must be either ‘Filter’ or ‘Indicator’.\n\n\nlabel\nThis is the version of the column name that the users will see on the platform, therefore you must fill this in and not leave it blank. For example, pupil_headcount may be ‘Number of pupil enrolments’. You should aim to keep these short and descriptive, but have the freedom to decide what is best to do for your users.\n\n\nindicator_grouping\nThis column gives production teams the option to add subheadings to group indicators in order to benefit the user. If this column is left blank, all indicators will be presented as one list of individual square radio boxes with no subheadings.\n\n\nindicator_unit\nIf this column is left blank then this will be a number by default, alternatively you can use either of the following units for financial or percentage measures - “£”, “£m”, “%”, “pp”.\n\n\nindicator_dp\nThis column allows you to set decimal place formatting for each of your indicators. If you leave it blank the platform will default to 2 d.p\n\n\nfilter_hint\nThis column gives you the option to add in a hint such as ‘Filter by school type’ for the filter to make the service easier for the users to navigate. If you leave the column blank, no hint will appear. Do not duplicate the column name here, as it will just appear twice\n\n\nfilter_grouping_column\nThis column should be blank unless you are wanting to group your filters. When you are wanting to group your filters this column should contain the exact name of the column/variable that you wish to group by. It is good practice to use the same variable name as that you are grouping, with _group appended at the end, i.e. ‘filter’ and ‘filter_group’\n\n\n\nNote that if you are using percentage points (pp) you must include a clear explanation in your release and methodology, so that users can understand what you are referring to."
- },
- {
- "objectID": "statistics-production/ud.html#example-ees-metadata",
- "href": "statistics-production/ud.html#example-ees-metadata",
- "title": "Open Data Standards",
- "section": "Example EES metadata",
- "text": "Example EES metadata\nEach row represents a column in the data file.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ncol_name\ncol_type\nlabel\nindicator_grouping\nindicator_unit\nindicator_dp\nfilter_hint\nfilter_grouping_column\n\n\n\n\ngender\nFilter\nGender\n\n\n\nFilter by pupil gender\n\n\n\nschool_phase\nFilter\nSchool phase\n\n\n\nFilter by the phase of the school\n\n\n\nnumber_children\nIndicator\nNumber of children\n\n\n\n\n\n\n\npercent_children\nIndicator\nPercentage of children\n\n%\n1\n\n\n\n\n\n\n\n\nThe corresponding data file.\n\n\n\n\n\n\n\n\n\n\n\ntime_period\n…\ncountry_name\ngender\nschool_phase\nnumber_children\npercent_children\n\n\n\n\n2018\n…\nEngland\nMale\nPrimary\n240\n26.7\n\n\n2018\n…\nEngland\nFemale\nPrimary\n200\n22.2\n\n\n2018\n…\nEngland\nTotal\nPrimary\n440\n48.9\n\n\n2018\n…\nEngland\nMale\nSecondary\n240\n26.7\n\n\n2018\n…\nEngland\nFemale\nSecondary\n220\n24.4\n\n\n2018\n…\nEngland\nTotal\nSecondary\n460\n51.1\n\n\n2018\n…\nEngland\nMale\nTotal\n480\n53.3\n\n\n2018\n…\nEngland\nFemale\nTotal\n420\n46.7\n\n\n2018\n…\nEngland\nTotal\nTotal\n900\n100.0"
- },
- {
- "objectID": "statistics-production/ud.html#time-columns",
- "href": "statistics-production/ud.html#time-columns",
- "title": "Open Data Standards",
- "section": "Time columns",
- "text": "Time columns\nWe use the two columns, time_period and time_identifier, to generalise time across our underlying datasets. All data files must contain these. This is a important for general useability of our data, as well as being critical in driving the charts and tables in the Explore Education Statistics platform and making explicit reference to the time in which our measurements relate to. This is a compulsory element of any official statistics dataset.\nIf you think that your data can’t follow this format, please contact statistics.development@education.gov.uk with details so that we can discuss this.\n\ntime_period must contain either a four digit year, or a 6 digit year.\ntime_period must be numeric. This allows the platform to understand ranges and order periods in a logical manner.\nsix digit time_periods must represent consecutive years - e.g. 201718, not 201619.\nIf you’re referring to a single term you should use the academic year not the calendar year in the time_period column.\nConceptually different years cannot be mixed in the same dataset.\nConceptually different year breakdowns (e.g. term, quarter, month), can be mixed with a full year of the same type using a filter column.\n\n\n\nSpecific time standards\n\nProducers should not mix different types of years in the same dataset. This is to prevent any chance of confusion for users selecting time periods with similar labels in the table tool. For example, you cannot have Academic year and Calendar year data in the same data file. You also cannot mix yearly breakdowns (e.g. full year, quarters, months, or terms) in the time identifier column. Instead, where it makes sense to mix these within a data file you should use a filter column as shown below. Note the use of ‘Total’, this is a part of the standards for filters.\n\n\n\ntime_period\ntime_identifier\nquarter\n\n\n\n\n201718\nAcademic year\nTotal\n\n\n201718\nAcademic year\nQ1\n\n\n201718\nAcademic year\nQ1-2\n\n\n\n\n\n\ntime_period\ntime_identifier\nmonth\n\n\n\n\n2017\nCalendar year\nTotal\n\n\n2017\nCalendar year\nJuly\n\n\n\nIf your row of data spans multiple years (e.g. is a cumulative sum between 2010 and 2018), the starting year should be made clear in the name of the indicator, with the year of the end of the time period listed as the time identifier. For example if you had been recording the number of enrolments in a Local authority since from the start of the 2010/11 Academic year to the end of the 2017/18 Academic year, your data would look like the example on the right.\n\n\n\ntime_period\ntime_identifier\nstarts_since_201011\n\n\n\n\n201718\nAcademic year\n190\n\n\n201617\nAcademic year\n173\n\n\n\n\n\n\nList of allowable time values\n\nAll time_period values should be numeric only, below the number of digits (either 4 or 6) is defined per time_identifier below. Do not include dashes or slashes in six digit years.\nYou can only mix time_identifiers if they appear within the same table below. If they are in separate tables then they should not be mixed.\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nCalendar year\n4 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nReporting year\n4 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nAcademic year\n6 digits\n\n\n\n\n\n\nacceptable time_identifier value\nCorresponding time_period\n\n\n\n\nFinancial year\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nFinancial year Q1\n6 digits\n\n\nFinancial year Q2\n6 digits\n\n\nFinancial year Q3\n6 digits\n\n\nFinancial year Q4\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nPart 1 (April to September)\n6 digits\n\n\nPart 2 (October to March)\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nTax year\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nAutumn term\n6 digits\n\n\nSpring term\n6 digits\n\n\nSummer term\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nAutumn and spring term\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nJanuary\n4 digits\n\n\nFebruary\n4 digits\n\n\nMarch\n4 digits\n\n\nApril\n4 digits\n\n\nMay\n4 digits\n\n\nJune\n4 digits\n\n\nJuly\n4 digits\n\n\nAugust\n4 digits\n\n\nSeptember\n4 digits\n\n\nOctober\n4 digits\n\n\nNovember\n4 digits\n\n\nDecember\n4 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nWeek 1\n4 digits\n\n\nWeek …\n4 digits\n\n\nWeek 52\n4 digits\n\n\n\n\n\n\n\nRemember\n\n\n\nYou must include time_period and time_identifier columns in your data files.\nYour data must match the allowable values above.\nUse ‘Reporting year’ if your data does not fit in other categories, i.e. collected on a specific day.\nIf you have different types of year such academic, calendar, and financial, these should be in separate files.\nUse filters to add more detail if you have multiple time breakdowns in the same file (quarter/full year).\nWhere a measure spans multiple years, you should name the starting year, and set the time_period as the year published."
- },
- {
- "objectID": "statistics-production/ud.html#geography-columns",
- "href": "statistics-production/ud.html#geography-columns",
- "title": "Open Data Standards",
- "section": "Geography columns",
- "text": "Geography columns\nWe publish at a number of different geography breakdowns and these vary from publication to publication. Every publication in the new platform must include the three compulsory geography columns - geographic_level, country_code and country_name in its data files. These are compulsory as the data we are producing must lie within a country boundary.\nThe geographic_level column should describe the level of data present in that row. Therefore data for a collection from a specific local authority would have ‘Local authority’ as the geographic_level, while a National aggregation would have ‘National’ as the geographic_level.\nTeams should make sure that they are regularly checking their geography codes if they are not using a lookup from a maintained database (such as in the PDR). ONS have the Open Geography portal, which can be a useful way of checking these. There is a wealth of data on there, though Local authority boundaries can be hard to find, they can be found using the tabs at the top – Boundaries > Administrative Boundaries > Counties and Unitary Authorities.\nTeams can also use the lookup tables available via the data screener GitHub repository data folder. Guidance on how to link directly to the latest data in your code can be seen in the standardised reference data section\nIf you have data from an unknown location, the standard is to use ‘Not available’ as name, and ‘x’ as the code/s, this clearly marks that the geographical data for that row is unavailable, and does so in a consistent way with the wider GSS. If you have a unique variation of a certain location, e.g. ‘Outside of England and unknown’, we may be able to add these in as exceptions. Locations like this would have a code of ‘z’ as no widely used code is applicable, please get in touch with us to discuss it further if you think this might apply to you.\nWe expect that all geography codes will be based on the Open Geography Portal and run checks against this for various levels in the data screener, please contact us if you have data that doesn’t match this.\n\nWhere you have data for a legacy LA that does not have a 9-digit new code, leave those cells as blank instead.\n\n\n\nDifferent measures of geography\n\nWhen using geographies that can be measured in multiple ways, you can achieve this by including a filter such as level_methodology in the example below to state how you have measured the geographic level. For example, at Local authority level you may have data that was measured by the residence of the pupil and the location of the school:\n\n\n\n\n\n\n\n\n\n\n\ngeographic_level\nold_la_code\nla_name\nnew_la_code\nlevel_methodology\nheadcount\n\n\n\n\nLocal authority\n373\nSheffield\nE08000019\nPupil residence\n689\n\n\nLocal authority\n373\nSheffield\nE08000019\nSchool location\n567\n\n\n\n\n\n\nAllowable geographic levels\n\nAll rows must have country_code and country_name completed, regardless of geographic level. The additional required columns by level are shown below. You do not have to publish at every level, this is a guide that covers every level that can be published in the platform.\nWhere you have multiple geographic levels in a file, leave any not applicable columns blank for other rows. For example, region_name and region_code should be blank for national rows. Some levels do fit into a hierarchy, for example local authorities all have a region, in these cases you should include the higher level information too. So for a local authority row, the region and country columns would all be completed as well as the la columns.\n\n\n\n\n\n\n\n\ngeographic_level\nrequired columns\nnotes\n\n\n\n\nNational\nNo additional columns\n\n\n\nRegional\nregion_code, region_name\n\n\n\nLocal authority\nold_la_code, new_la_code, la_name\nIt is usually good practice to include the Regional aggregations where possible given the direct link between Local authorities and Regions.\n\n\nRSC region\nrsc_region_lead_name\nFor RSC region data, we generally define them into lead RSC regions where the majority of the data is from.\n\n\nParliamentary constituency\npcon_code, pcon_name\n\n\n\nLocal authority district\nlad_code, lad_name\n\n\n\nLocal enterprise partnership\nlocal_enterprise_partnership_code, local_enterprise_partnership_name\n\n\n\nEnglish devolved area\nenglish_devolved_area_code, english_devolved_area_name\n\n\n\nOpportunity area\nopportunity_area_code, opportunity_area_name\n\n\n\nWard\nward_code, ward_name\n\n\n\nMAT\ntrust_id, trust_name\nNote that Trust ID is shown as Group ID on GIAS when looking at a Trust. MATs also have a ‘company number’, this can be included but is not mandatory.\n\n\nSponsor\nsponsor_id, sponsor_name\nNote that Sponsor ID is shown as Group ID on GIAS when looking at a Sponsor.\n\n\n\n\nEnglish devolved area is used to refer to combined authorities, mayoral combined authorities and the Greater London Authority.\n\nPlanning area, School, Provider, and Institution level data will upload as normal to EES, though will not be read into the table tool or data blocks if they are mixed in with other levels. All data, including these levels are accessible in the downloadable files for users to explore in the same format as they are uploaded.\nFor Provider, any data files that only consist of this level are able to be uploaded and will be usable in the table tool. This solution is still not ideal, though it will allow you to include additional filters at provider level. Wherever there are multiple values per provider this should be marked as a filter, and where there is a single value per provider this should be marked as an indicator.\nFor School, any files consisting of school-level data should be standalone data files (that will work in the table tool, not ancillary files) and these should all be passing the EES data screener. They should also include LA level information so that users can see this when searching.\n\n\n\n\n\n\n\n\ngeographic_level\nrequired columns\nnotes\n\n\n\n\nSchool\nschool_name, school_urn, school_laestab, old_la_code, new_la_code, la_name\n\n\n\nProvider\nprovider_name, provider_ukprn\n\n\n\n\nFor Planning area and Institution data, any data files that only consist of these levels should be uploaded as an ancillary file rather than a data file.\n\n\n\n\n\n\n\n\ngeographic_level\nrequired columns\nnotes\n\n\n\n\nPlanning area\nNo required columns, though we recommend both planning_area_code and planning_area_name\n\n\n\nInstitution\nNo required columns, though we recommend you include institution_id and institution_name to make data matching easier\n\n\n\n\n\nIf you have a level that isn’t covered above, please contact statistics.development@education.gov.uk with details and example data.\n\n\n\n\nUsing School or Provider as a filter\n\nProvider / School breakdowns can work as a filter column in your data file but only if it is the only filter in the file. We’d recommend using school_name or provider_name as the filter, and then including the code as well, such as Malton School (URN: 121681) or Malton School (UKPRN: 10004165). You should also include LA level information so that users can see this when searching.\nThis approach will work for basic filtering of providers and schools but it may not be performant in the table tool and the number of choices made available to users may be overwhelming (especially for schools, or data set to national level).\n\n\n\nUnknown geographic codes\n\n\n\n\n\n\n\n\n\nGeographical code\nUsage\nExample\n\n\n\n\nz\nWhen a geography code is not applicable\nUsing custom geographies like “Scotland, Wales and Northern Ireland” as a group. This has no standardised geography code so will not be applicable\n\n\nx\nWhen a geography code is unavailable\nData is missing, the geography is unknown so no code is available\n\n\n\n\n\n\n\nRemember\n\n\n\nYou must have the minimum expected geography columns, country_code and country_name.\nYou must also have the additional columns required for all geographies included in your file.\nThe additional columns must exactly match the names above.\nYou do not have to publish at every level in the guide, publish the selection of levels you feel appropriate.\nYou should regularly check that you are using the most up to date geography names and codes.\nAside from blanks, no two geographic locations should have the same code.\nYou can mix geographies in the same file, and should avoid separating files by geography unless size is an issue."
- },
- {
- "objectID": "statistics-production/ud.html#ethnicity",
- "href": "statistics-production/ud.html#ethnicity",
- "title": "Open Data Standards",
- "section": "Ethnicity",
- "text": "Ethnicity\n\nOverview\n\nRace is a protected characteristic under the Equality Act 2010 and can include colour, nationality, ethnic or national origins. Data collected on ethnicity is self-identified and captures a person’s feelings of cultural heritage and belonging. Everyone may have slightly different ideas and feelings about what ethnicity encompasses, race and nationality may, or may not, be a part of a persons’ self-identified ethnicity.\nThis guidance has been written by collating the most up to date advice and guidance from the following sources:\n\nGovernment Statistical Service (GSS)\nOffice for National Statistics\nCabinet Office\nCommission on Race and Ethnic Disparities (CRED) report\n\n\n\n\nGSS ethnicity categories\n\nEthnicity breakdowns should be presented within the standard field names in any data files containing ethnicity breakdowns: ethnicity_major, ethnicity_minor or ethnicity_detailed. The first two are outlined in reference to the GSS guidelines on ethnic groupings below, whilst the third is for any ethnicity fields containing finer grained breakdowns than described in ethnicity_minor below.\nThe GSS publish standards on how to collect ethnicity data based on research conducted for the UK Census. The current guidelines (shown below) were developed as part of the 2011 Census and were unchanged in the 2021 census. We aim to follow as closely as reasonable to the GSS standards and guidance. Note the use of spaces around forward-slashes, which allows us to meet accessibility standards (in particular for the use of screen readers) as well as improving automated wrapping in text and value boxes.\n\n\n\n\n\n\n\nethnicity_major\nethnicity_minor\n\n\n\n\nWhite\nEnglish / Welsh / Scottish / Northern Irish / British\n\n\n\nIrish\n\n\n\nGypsy or Irish Traveller\n\n\n\nAny other White background\n\n\nMixed / Multiple ethnic groups\nWhite and Black Caribbean\n\n\n\nWhite and Black African\n\n\n\nWhite and Asian\n\n\n\nAny other Mixed / Multiple ethnic background\n\n\nAsian / Asian British\nIndian\n\n\n\nPakistani\n\n\n\nBangladeshi\n\n\n\nChinese\n\n\n\nAny other Asian background\n\n\nBlack / African / Caribbean / Black British\nAfrican\n\n\n\nCaribbean\n\n\n\nAny other Black / African / Caribbean background\n\n\nOther ethnic group\nArab\n\n\n\nAny other ethnic group\n\n\nUnknown\nUnknown\n\n\n\nNote that these standards are written from the perspective of creating survey questions, i.e. the data collection phase. Where teams are trying to aggregate existing categories up to match the GSS guidance, careful consideration of how and where differing systems may or may not match with the guidance is necessary.\nWe recommend two alternative methods of ordering the above ethnic groups when reporting on ethnicity statistics:\n\nAlphabetical: use in tables and when listing ethnic groups (with ‘Other ethnic group’ and sometimes ‘Unknown’ as a final category)\nIn expected order of size (with largest first): useful in charts and visualisations as it makes data and patterns easier to read\n\n\n\n\nReporting on broad ethnic minorities catageories (e.g. BAME)\n\nPublished in March 2021, the report of the Commission on Race and Ethnic Disparities made a number of suggestions on how we discuss and report on race in the UK.\nOne of the major elements was around aggregation of ethnic minorities into a single over-arching group (in particular BAME):\n\nRecommendation 24: Disaggregate the term ‘BAME’ Stop using aggregated and unhelpful terms such as ‘BAME’, to better focus on understanding disparities and outcomes for specific ethnic groups.\n\nThe reasons for this approach are two-fold:\n\nGrouping separate ethnic minorities into a single over-arching group can mask differing trends that may be present in the underlying data for groups experiencing different pressures and environments within UK society.\nThe phrasing ‘Black and Minority Ethnic’ and ‘Black, Asian and Minority Ethnic’ give an emphasis to specific ethnic groups whilst excluding others. This could be misleading or diminish consideration of the unnamed groups within the statistics.\n\nBased on this guidance, the following is how teams should aggregate and write about ethnic minority groups.\nStatistics producers should avoid where possible the practice of reporting aggregates grouping together disparate ethnic groups. Instead, statistics should be reported individually for the recommended top-level groups:\n\nWhite\nMixed / Multiple ethnic groups\nAsian / Asian British\nBlack / African / Caribbean / Black British\nOther ethnic group\n\nWhere there is a reasonable need to publish or report statistics for a full aggregation of ethnic minorities (such as reporting on existing historical targets), official guidance is as follows:\n\nUse ‘ethnic minorities’ to refer to all ethnic groups except the white British group. This term includes white minorities, such as Gypsy, Roma and Irish Traveller groups. For comparisons with the white group as a whole, use ‘ethnic minorities (excluding White minorities)’."
- },
- {
- "objectID": "statistics-production/pub.html",
- "href": "statistics-production/pub.html",
- "title": "Publication checklists",
- "section": "",
- "text": "Guidance for how to publish different types of statistics"
- },
- {
- "objectID": "statistics-production/pub.html#how-to-publish",
- "href": "statistics-production/pub.html#how-to-publish",
- "title": "Publication checklists",
- "section": "How to publish",
- "text": "How to publish\n\n\n\nEES create release one pager"
- },
- {
- "objectID": "statistics-production/pub.html#publication-checklist",
- "href": "statistics-production/pub.html#publication-checklist",
- "title": "Publication checklists",
- "section": "Publication checklist",
- "text": "Publication checklist\nBefore releasing statistics for the first time you may want to discuss the new process with key stakeholders and/or pre-release users to make them aware of the new service. You should also inform the Explore Statistics Mailbox and HoP Office teams.\nBefore you start creating a release in the platform you should have:\n\nAnnounced the upcoming release via gov.uk\nSent metadata form to HoP\nContacted the BAU team so we can support you with your first release\nProduced your tidy csv data files with appropriate disclosure control\nProduced metadata files for each csv data file\nRan your data and metadata through our screener checks\n\nBefore you publish a release you have created in the platform you should have:\n\nChecked all the data has loaded successfully\nWritten footnotes\nWritten content (including tables and charts)\nCreated a data guidance document\nCreated a public pre-release access list\nEnsured methodology information is either linked off to or attached to the release\nPassed the release for higher review (senior sign-off)\nPreviewed your release\nScheduled the release date\nInvited your PRA list to preview 24 hours before it goes live\nRaised a web ticket for the associated gov.uk page\n\n\nWord templates for the data guidance, pra-list, and content can be found on sharepoint."
- },
- {
- "objectID": "statistics-production/pub.html#linking-to-gov.uk",
- "href": "statistics-production/pub.html#linking-to-gov.uk",
- "title": "Publication checklists",
- "section": "Linking to gov.uk",
- "text": "Linking to gov.uk\nYou will need to arrange a gov.uk statistics publication page so that it links to EES. Here is how to do that:\nTwo days ahead of publication, you’ll need to raise a ticket with the Digital communications (gov.uk) team and ask them to create a new gov.uk statistics page with a link to EES, connect it to the announcement and add to any collections.\nIn your request, you’ll need to include:\n\ntitle, summary sentence and ‘detail’ for the new page – you can include a link to previous releases if you want it to be the same\nthe link for your EES release – if you don’t have it you can update the ticket when the link is available\nthe link for the announcement\nthe link of any gov.uk collections it needs to be added to\nan email with clearance from your deputy director\nan email confirming communications are happy for it to go if it’s for stats that aren’t pre-announced or it’s an update to stats made after publication – so that they can prepare reactive lines\n\nHere are examples of how the page will look like:\n\nPrimary School Performance Tables 2018\nHigher Education Student Statistics 2018 to 2019\n\n\nYou can find what the link to your EES release will be by looking at the ‘Sign off’ page within the release dashboard on EES.\n\n\nText to update the collection page:\nIn addition, you will also need to raise a ticket with the Digital communications (gov.uk) team to update your collection page/s with the following text:\n\nFrom [publication date], the Department for Education will be publishing the [publication title] on a new statistics site, Explore Education Statistics (EES).\n\nWhen you raise the ticket please ensure that all the documents you submit are in an accessible format as stated in the accessibility guidance. There is also guidance on accessibility for Excel workbooks in Teams."
- },
- {
- "objectID": "statistics-production/pub.html#how-to-publish-1",
- "href": "statistics-production/pub.html#how-to-publish-1",
- "title": "Publication checklists",
- "section": "How to publish",
- "text": "How to publish\nAhead of publication, you’ll need to raise a ticket with the Digital communications (gov.uk) team and ask them to create a new release (or add to a series if this already exists).\nIn your request, you’ll need to include:\n\ntitle, summary sentence and ‘detail’ for the new page\nthe link for the announcement\nthe link of any gov.uk collections it needs to be added to\nan attachment with the accessible data files\nan email with clearance from your deputy director\n\nWhen you raise the ticket please ensure that all the documents you submit are in an accessible format as stated in the accessibility guidance. There is also guidance on accessibility for Excel workbooks in Teams."
- },
- {
- "objectID": "statistics-production/pub.html#publication-checklist-1",
- "href": "statistics-production/pub.html#publication-checklist-1",
- "title": "Publication checklists",
- "section": "Publication checklist",
- "text": "Publication checklist\nBefore you start creating your release you should have:\n\nAnnounced the upcoming release via gov.uk\nSent metadata form to HoP\n\nBefore you publish a release you should have:\n\nChecked that the data meets accessibility requirements, ideally as a .csv file, or an accessible Excel file where this is not possible\nWritten footnotes where appropriate\nPassed the release for higher review (senior sign-off)\nSent the final files and sign-off email to the digital communications (gov.uk) team"
+ "title": "Open Data Standards",
+ "section": "File size",
+ "text": "File size\nThere are no character or size limits in a csv file and there is no size limit for EES, though the larger a file is, the longer it will take to upload and process. Also remember that the files you upload are the files that users will download, consider the software they may access to (e.g. Excel) and whether the size of your files are compatible with this.\nExcel has a cell character limit of 32,760 and a row limit of 1,048,576. It is best to avoid exceeding these as some end users may struggle to open the file. One good way to cut the file down is to split after a certain number of years, or to separate out different geographic levels into separate files, providing school level data as a separate file for example. With the data all being in a tidy format these are then easy enough for secondary analysts to stitch back together if needed.\nA rough guide to file size would be:\n\nAnything under 10mb is relatively small\n10mb to 100mb is a fairly common file size that most teams have\n100mb to 500mb is a large file and will struggle to upload if not compressed to a zip folder.\n500mb and over are very large, and sometimes may struggle to compress small enough to upload.\n4gb or more in size is larger than any we have seen before and will likely need testing in the platform first.\n\nContact us if you have any issues, or files that might be over 1gb."
},
{
- "objectID": "statistics-production/pub.html#how-to-publish-2",
- "href": "statistics-production/pub.html#how-to-publish-2",
- "title": "Publication checklists",
- "section": "How to publish",
- "text": "How to publish\nWe recommend that teams use R Shiny to build and publish dashboards. Shiny applications are easily made accessible, and underlying code for Shiny apps can be published alongside for transparency.\nFor more information on publishing dashboards, including template code and how to get started, see our Public dashboards page."
+ "objectID": "statistics-production/ud.html#data-symbols",
+ "href": "statistics-production/ud.html#data-symbols",
+ "title": "Open Data Standards",
+ "section": "Data symbols",
+ "text": "Data symbols\nIn line with the GSS guidance on symbols, special values should be replaced with symbols in the following situations:\n\n\n\n\n\n\n\n\n\nSymbol\nUsage\nExample\nObsolete equivalents\n\n\n\n\nz\nWhen an observation is not applicable\nNo data for at gender level for boys at an all-girls school\n\n\n\nx\nWhen data is unavailable for other reasons\nData for an indicator is not collected in a certain region\n:\n\n\nc\nConfidential data\nData has been suppressed\n\n\n\nlow\nRounds to 0, but is not 0\nRounding to the nearest thousand, 499 would otherwise show as 0. Only use 0 for true 0’s\n~\n\n\nu\nWhen an observation is of low reliability\nData for a local authority is identified as missing returns so is removed from regional and national totals\n\n\n\n\nIf you have any other conventions you’ve used in previous publications, or a scenario that isn’t covered above, check the GSS guidance (ignoring the part around separate columns for symbols), and contact us."
},
{
- "objectID": "statistics-production/embedded-charts.html",
- "href": "statistics-production/embedded-charts.html",
- "title": "Embedded visualisations in EES",
- "section": "",
- "text": "Guidance for creating and embedding R-Shiny visualisations in EES publications"
+ "objectID": "statistics-production/ud.html#mandatory-ees-metadata-columns",
+ "href": "statistics-production/ud.html#mandatory-ees-metadata-columns",
+ "title": "Open Data Standards",
+ "section": "Mandatory EES metadata columns",
+ "text": "Mandatory EES metadata columns\n\n\n\n\n\n\n\ncolumn\ndetails\n\n\n\n\ncol_name\nThis must exactly match the name of the corresponding column in the dataset.\n\n\ncol_type\nThis must be either ‘Filter’ or ‘Indicator’.\n\n\nlabel\nThis is the version of the column name that the users will see on the platform, therefore you must fill this in and not leave it blank. For example, pupil_headcount may be ‘Number of pupil enrolments’. You should aim to keep these short and descriptive, but have the freedom to decide what is best to do for your users.\n\n\nindicator_grouping\nThis column gives production teams the option to add subheadings to group indicators in order to benefit the user. If this column is left blank, all indicators will be presented as one list of individual square radio boxes with no subheadings.\n\n\nindicator_unit\nIf this column is left blank then this will be a number by default, alternatively you can use either of the following units for financial or percentage measures - “£”, “£m”, “%”, “pp”.\n\n\nindicator_dp\nThis column allows you to set decimal place formatting for each of your indicators. If you leave it blank the platform will default to 2 d.p\n\n\nfilter_hint\nThis column gives you the option to add in a hint such as ‘Filter by school type’ for the filter to make the service easier for the users to navigate. If you leave the column blank, no hint will appear. Do not duplicate the column name here, as it will just appear twice\n\n\nfilter_grouping_column\nThis column should be blank unless you are wanting to group your filters. When you are wanting to group your filters this column should contain the exact name of the column/variable that you wish to group by. It is good practice to use the same variable name as that you are grouping, with _group appended at the end, i.e. ‘filter’ and ‘filter_group’\n\n\n\nNote that if you are using percentage points (pp) you must include a clear explanation in your release and methodology, so that users can understand what you are referring to."
},
{
- "objectID": "statistics-production/embedded-charts.html#when-to-use-an-embedded-chart",
- "href": "statistics-production/embedded-charts.html#when-to-use-an-embedded-chart",
- "title": "Embedded visualisations in EES",
- "section": "When to use an embedded chart",
- "text": "When to use an embedded chart\nEES provides a wide range of inbuilt chart options and will always be the first preference for static line, bar and geographical charts. This helps us to clearly maintain consistent styling and accessibility levels across the site.\nHowever, there are some instances where you might want to publish something that we can’t provide through EES. Example use cases would be:\n\ninteractive charts controlled by drop-down filters;\nchart types not provided by EES, e.g. sankey diagrams, box plots, waffle charts, pie charts, dumbbell charts, etc."
+ "objectID": "statistics-production/ud.html#example-ees-metadata",
+ "href": "statistics-production/ud.html#example-ees-metadata",
+ "title": "Open Data Standards",
+ "section": "Example EES metadata",
+ "text": "Example EES metadata\nEach row represents a column in the data file.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ncol_name\ncol_type\nlabel\nindicator_grouping\nindicator_unit\nindicator_dp\nfilter_hint\nfilter_grouping_column\n\n\n\n\ngender\nFilter\nGender\n\n\n\nFilter by pupil gender\n\n\n\nschool_phase\nFilter\nSchool phase\n\n\n\nFilter by the phase of the school\n\n\n\nnumber_children\nIndicator\nNumber of children\n\n\n\n\n\n\n\npercent_children\nIndicator\nPercentage of children\n\n%\n1\n\n\n\n\n\n\n\n\nThe corresponding data file.\n\n\n\n\n\n\n\n\n\n\n\ntime_period\n…\ncountry_name\ngender\nschool_phase\nnumber_children\npercent_children\n\n\n\n\n2018\n…\nEngland\nMale\nPrimary\n240\n26.7\n\n\n2018\n…\nEngland\nFemale\nPrimary\n200\n22.2\n\n\n2018\n…\nEngland\nTotal\nPrimary\n440\n48.9\n\n\n2018\n…\nEngland\nMale\nSecondary\n240\n26.7\n\n\n2018\n…\nEngland\nFemale\nSecondary\n220\n24.4\n\n\n2018\n…\nEngland\nTotal\nSecondary\n460\n51.1\n\n\n2018\n…\nEngland\nMale\nTotal\n480\n53.3\n\n\n2018\n…\nEngland\nFemale\nTotal\n420\n46.7\n\n\n2018\n…\nEngland\nTotal\nTotal\n900\n100.0"
},
{
- "objectID": "statistics-production/embedded-charts.html#tools",
- "href": "statistics-production/embedded-charts.html#tools",
- "title": "Embedded visualisations in EES",
- "section": "Tools",
- "text": "Tools\nWe currently only support custom charts created using R-Shiny. These should be created with ggplot and plotly. We provide a template example of a demo R-Shiny/ggplot chart on the DfE Analytical Services GitHub site, which is described below."
+ "objectID": "statistics-production/ud.html#time-columns",
+ "href": "statistics-production/ud.html#time-columns",
+ "title": "Open Data Standards",
+ "section": "Time columns",
+ "text": "Time columns\nWe use the two columns, time_period and time_identifier, to generalise time across our underlying datasets. All data files must contain these. This is a important for general useability of our data, as well as being critical in driving the charts and tables in the Explore Education Statistics platform and making explicit reference to the time in which our measurements relate to. This is a compulsory element of any official statistics dataset.\nIf you think that your data can’t follow this format, please contact statistics.development@education.gov.uk with details so that we can discuss this.\n\ntime_period must contain either a four digit year, or a 6 digit year.\ntime_period must be numeric. This allows the platform to understand ranges and order periods in a logical manner.\nsix digit time_periods must represent consecutive years - e.g. 201718, not 201619.\nIf you’re referring to a single term you should use the academic year not the calendar year in the time_period column.\nConceptually different years cannot be mixed in the same dataset.\nConceptually different year breakdowns (e.g. term, quarter, month), can be mixed with a full year of the same type using a filter column.\n\n\n\nSpecific time standards\n\nProducers should not mix different types of years in the same dataset. This is to prevent any chance of confusion for users selecting time periods with similar labels in the table tool. For example, you cannot have Academic year and Calendar year data in the same data file. You also cannot mix yearly breakdowns (e.g. full year, quarters, months, or terms) in the time identifier column. Instead, where it makes sense to mix these within a data file you should use a filter column as shown below. Note the use of ‘Total’, this is a part of the standards for filters.\n\n\n\ntime_period\ntime_identifier\nquarter\n\n\n\n\n201718\nAcademic year\nTotal\n\n\n201718\nAcademic year\nQ1\n\n\n201718\nAcademic year\nQ1-2\n\n\n\n\n\n\ntime_period\ntime_identifier\nmonth\n\n\n\n\n2017\nCalendar year\nTotal\n\n\n2017\nCalendar year\nJuly\n\n\n\nIf your row of data spans multiple years (e.g. is a cumulative sum between 2010 and 2018), the starting year should be made clear in the name of the indicator, with the year of the end of the time period listed as the time identifier. For example if you had been recording the number of enrolments in a Local authority since from the start of the 2010/11 Academic year to the end of the 2017/18 Academic year, your data would look like the example on the right.\n\n\n\ntime_period\ntime_identifier\nstarts_since_201011\n\n\n\n\n201718\nAcademic year\n190\n\n\n201617\nAcademic year\n173\n\n\n\n\n\n\nList of allowable time values\n\nAll time_period values should be numeric only, below the number of digits (either 4 or 6) is defined per time_identifier below. Do not include dashes or slashes in six digit years.\nYou can only mix time_identifiers if they appear within the same table below. If they are in separate tables then they should not be mixed.\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nCalendar year\n4 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nReporting year\n4 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nAcademic year\n6 digits\n\n\n\n\n\n\nacceptable time_identifier value\nCorresponding time_period\n\n\n\n\nFinancial year\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nFinancial year Q1\n6 digits\n\n\nFinancial year Q2\n6 digits\n\n\nFinancial year Q3\n6 digits\n\n\nFinancial year Q4\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nPart 1 (April to September)\n6 digits\n\n\nPart 2 (October to March)\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nTax year\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nAutumn term\n6 digits\n\n\nSpring term\n6 digits\n\n\nSummer term\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nAutumn and spring term\n6 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nJanuary\n4 digits\n\n\nFebruary\n4 digits\n\n\nMarch\n4 digits\n\n\nApril\n4 digits\n\n\nMay\n4 digits\n\n\nJune\n4 digits\n\n\nJuly\n4 digits\n\n\nAugust\n4 digits\n\n\nSeptember\n4 digits\n\n\nOctober\n4 digits\n\n\nNovember\n4 digits\n\n\nDecember\n4 digits\n\n\n\n\n\n\nAcceptable time_identifier value\nCorresponding time_period\n\n\n\n\nWeek 1\n4 digits\n\n\nWeek …\n4 digits\n\n\nWeek 52\n4 digits\n\n\n\n\n\n\n\nRemember\n\n\n\nYou must include time_period and time_identifier columns in your data files.\nYour data must match the allowable values above.\nUse ‘Reporting year’ if your data does not fit in other categories, i.e. collected on a specific day.\nIf you have different types of year such academic, calendar, and financial, these should be in separate files.\nUse filters to add more detail if you have multiple time breakdowns in the same file (quarter/full year).\nWhere a measure spans multiple years, you should name the starting year, and set the time_period as the year published."
},
{
- "objectID": "statistics-production/embedded-charts.html#review-and-authorisation",
- "href": "statistics-production/embedded-charts.html#review-and-authorisation",
- "title": "Embedded visualisations in EES",
- "section": "Review and authorisation",
- "text": "Review and authorisation\nTo get a custom chart approved for embedding within a publication, you’ll need to get it reviewed by the Statistics Development team (in addition to your standard approval chain)."
+ "objectID": "statistics-production/ud.html#geography-columns",
+ "href": "statistics-production/ud.html#geography-columns",
+ "title": "Open Data Standards",
+ "section": "Geography columns",
+ "text": "Geography columns\nWe publish at a number of different geography breakdowns and these vary from publication to publication. Every publication in the new platform must include the three compulsory geography columns - geographic_level, country_code and country_name in its data files. These are compulsory as the data we are producing must lie within a country boundary.\nThe geographic_level column should describe the level of data present in that row. Therefore data for a collection from a specific local authority would have ‘Local authority’ as the geographic_level, while a National aggregation would have ‘National’ as the geographic_level.\nTeams should make sure that they are regularly checking their geography codes if they are not using a lookup from a maintained database (such as in the PDR). ONS have the Open Geography portal, which can be a useful way of checking these. There is a wealth of data on there, though Local authority boundaries can be hard to find, they can be found using the tabs at the top – Boundaries > Administrative Boundaries > Counties and Unitary Authorities.\nTeams can also use the lookup tables available via the data screener GitHub repository data folder. Guidance on how to link directly to the latest data in your code can be seen in the standardised reference data section\nIf you have data from an unknown location, the standard is to use ‘Not available’ as name, and ‘x’ as the code/s, this clearly marks that the geographical data for that row is unavailable, and does so in a consistent way with the wider GSS. If you have a unique variation of a certain location, e.g. ‘Outside of England and unknown’, we may be able to add these in as exceptions. Locations like this would have a code of ‘z’ as no widely used code is applicable, please get in touch with us to discuss it further if you think this might apply to you.\nWe expect that all geography codes will be based on the Open Geography Portal and run checks against this for various levels in the data screener, please contact us if you have data that doesn’t match this.\n\nWhere you have data for a legacy LA that does not have a 9-digit new code, leave those cells as blank instead.\n\n\n\nDifferent measures of geography\n\nWhen using geographies that can be measured in multiple ways, you can achieve this by including a filter such as level_methodology in the example below to state how you have measured the geographic level. For example, at Local authority level you may have data that was measured by the residence of the pupil and the location of the school:\n\n\n\n\n\n\n\n\n\n\n\ngeographic_level\nold_la_code\nla_name\nnew_la_code\nlevel_methodology\nheadcount\n\n\n\n\nLocal authority\n373\nSheffield\nE08000019\nPupil residence\n689\n\n\nLocal authority\n373\nSheffield\nE08000019\nSchool location\n567\n\n\n\n\n\n\nAllowable geographic levels\n\nAll rows must have country_code and country_name completed, regardless of geographic level. The additional required columns by level are shown below. You do not have to publish at every level, this is a guide that covers every level that can be published in the platform.\nWhere you have multiple geographic levels in a file, leave any not applicable columns blank for other rows. For example, region_name and region_code should be blank for national rows. Some levels do fit into a hierarchy, for example local authorities all have a region, in these cases you should include the higher level information too. So for a local authority row, the region and country columns would all be completed as well as the la columns.\n\n\n\n\n\n\n\n\ngeographic_level\nrequired columns\nnotes\n\n\n\n\nNational\nNo additional columns\n\n\n\nRegional\nregion_code, region_name\n\n\n\nLocal authority\nold_la_code, new_la_code, la_name\nIt is usually good practice to include the Regional aggregations where possible given the direct link between Local authorities and Regions.\n\n\nRSC region\nrsc_region_lead_name\nFor RSC region data, we generally define them into lead RSC regions where the majority of the data is from.\n\n\nParliamentary constituency\npcon_code, pcon_name\n\n\n\nLocal authority district\nlad_code, lad_name\n\n\n\nLocal enterprise partnership\nlocal_enterprise_partnership_code, local_enterprise_partnership_name\n\n\n\nEnglish devolved area\nenglish_devolved_area_code, english_devolved_area_name\n\n\n\nOpportunity area\nopportunity_area_code, opportunity_area_name\n\n\n\nWard\nward_code, ward_name\n\n\n\nMAT\ntrust_id, trust_name\nNote that Trust ID is shown as Group ID on GIAS when looking at a Trust. MATs also have a ‘company number’, this can be included but is not mandatory.\n\n\nSponsor\nsponsor_id, sponsor_name\nNote that Sponsor ID is shown as Group ID on GIAS when looking at a Sponsor.\n\n\n\n\nEnglish devolved area is used to refer to combined authorities, mayoral combined authorities and the Greater London Authority.\n\nPlanning area, School, Provider, and Institution level data will upload as normal to EES, though will not be read into the table tool or data blocks if they are mixed in with other levels. All data, including these levels are accessible in the downloadable files for users to explore in the same format as they are uploaded.\nFor Provider, any data files that only consist of this level are able to be uploaded and will be usable in the table tool. This solution is still not ideal, though it will allow you to include additional filters at provider level. Wherever there are multiple values per provider this should be marked as a filter, and where there is a single value per provider this should be marked as an indicator.\nFor School, any files consisting of school-level data should be standalone data files (that will work in the table tool, not ancillary files) and these should all be passing the EES data screener. They should also include LA level information so that users can see this when searching.\n\n\n\n\n\n\n\n\ngeographic_level\nrequired columns\nnotes\n\n\n\n\nSchool\nschool_name, school_urn, school_laestab, old_la_code, new_la_code, la_name\n\n\n\nProvider\nprovider_name, provider_ukprn\n\n\n\n\nFor Planning area and Institution data, any data files that only consist of these levels should be uploaded as an ancillary file rather than a data file.\n\n\n\n\n\n\n\n\ngeographic_level\nrequired columns\nnotes\n\n\n\n\nPlanning area\nNo required columns, though we recommend both planning_area_code and planning_area_name\n\n\n\nInstitution\nNo required columns, though we recommend you include institution_id and institution_name to make data matching easier\n\n\n\n\n\nIf you have a level that isn’t covered above, please contact statistics.development@education.gov.uk with details and example data.\n\n\n\n\nUsing School or Provider as a filter\n\nProvider / School breakdowns can work as a filter column in your data file but only if it is the only filter in the file. We’d recommend using school_name or provider_name as the filter, and then including the code as well, such as Malton School (URN: 121681) or Malton School (UKPRN: 10004165). You should also include LA level information so that users can see this when searching.\nThis approach will work for basic filtering of providers and schools but it may not be performant in the table tool and the number of choices made available to users may be overwhelming (especially for schools, or data set to national level).\n\n\n\nUnknown geographic codes\n\n\n\n\n\n\n\n\n\nGeographical code\nUsage\nExample\n\n\n\n\nz\nWhen a geography code is not applicable\nUsing custom geographies like “Scotland, Wales and Northern Ireland” as a group. This has no standardised geography code so will not be applicable\n\n\nx\nWhen a geography code is unavailable\nData is missing, the geography is unknown so no code is available\n\n\n\n\n\n\n\nRemember\n\n\n\nYou must have the minimum expected geography columns, country_code and country_name.\nYou must also have the additional columns required for all geographies included in your file.\nThe additional columns must exactly match the names above.\nYou do not have to publish at every level in the guide, publish the selection of levels you feel appropriate.\nYou should regularly check that you are using the most up to date geography names and codes.\nAside from blanks, no two geographic locations should have the same code.\nYou can mix geographies in the same file, and should avoid separating files by geography unless size is an issue."
},
{
- "objectID": "statistics-production/embedded-charts.html#the-dfe-tiny-shiny-template",
- "href": "statistics-production/embedded-charts.html#the-dfe-tiny-shiny-template",
- "title": "Embedded visualisations in EES",
- "section": "The DfE Tiny-Shiny template",
- "text": "The DfE Tiny-Shiny template\nOur template tiny shiny app repository should be used a starting point for all embedded shiny charts.\nTo get an app set-up for use with EES, you’ll need the Statistics Development Team team to create a repo for the app within the DfE Analytical Services area on GitHub."
+ "objectID": "statistics-production/ud.html#ethnicity",
+ "href": "statistics-production/ud.html#ethnicity",
+ "title": "Open Data Standards",
+ "section": "Ethnicity",
+ "text": "Ethnicity\n\nOverview\n\nRace is a protected characteristic under the Equality Act 2010 and can include colour, nationality, ethnic or national origins. Data collected on ethnicity is self-identified and captures a person’s feelings of cultural heritage and belonging. Everyone may have slightly different ideas and feelings about what ethnicity encompasses, race and nationality may, or may not, be a part of a persons’ self-identified ethnicity.\nThis guidance has been written by collating the most up to date advice and guidance from the following sources:\n\nGovernment Statistical Service (GSS)\nOffice for National Statistics\nCabinet Office\nCommission on Race and Ethnic Disparities (CRED) report\n\n\n\n\nGSS ethnicity categories\n\nEthnicity breakdowns should be presented within the standard field names in any data files containing ethnicity breakdowns: ethnicity_major, ethnicity_minor or ethnicity_detailed. The first two are outlined in reference to the GSS guidelines on ethnic groupings below, whilst the third is for any ethnicity fields containing finer grained breakdowns than described in ethnicity_minor below.\nThe GSS publish standards on how to collect ethnicity data based on research conducted for the UK Census. The current guidelines (shown below) were developed as part of the 2011 Census and were unchanged in the 2021 census. We aim to follow as closely as reasonable to the GSS standards and guidance. Note the use of spaces around forward-slashes, which allows us to meet accessibility standards (in particular for the use of screen readers) as well as improving automated wrapping in text and value boxes.\n\n\n\n\n\n\n\nethnicity_major\nethnicity_minor\n\n\n\n\nWhite\nEnglish / Welsh / Scottish / Northern Irish / British\n\n\n\nIrish\n\n\n\nGypsy or Irish Traveller\n\n\n\nAny other White background\n\n\nMixed / Multiple ethnic groups\nWhite and Black Caribbean\n\n\n\nWhite and Black African\n\n\n\nWhite and Asian\n\n\n\nAny other Mixed / Multiple ethnic background\n\n\nAsian / Asian British\nIndian\n\n\n\nPakistani\n\n\n\nBangladeshi\n\n\n\nChinese\n\n\n\nAny other Asian background\n\n\nBlack / African / Caribbean / Black British\nAfrican\n\n\n\nCaribbean\n\n\n\nAny other Black / African / Caribbean background\n\n\nOther ethnic group\nArab\n\n\n\nAny other ethnic group\n\n\nUnknown\nUnknown\n\n\n\nNote that these standards are written from the perspective of creating survey questions, i.e. the data collection phase. Where teams are trying to aggregate existing categories up to match the GSS guidance, careful consideration of how and where differing systems may or may not match with the guidance is necessary.\nWe recommend two alternative methods of ordering the above ethnic groups when reporting on ethnicity statistics:\n\nAlphabetical: use in tables and when listing ethnic groups (with ‘Other ethnic group’ and sometimes ‘Unknown’ as a final category)\nIn expected order of size (with largest first): useful in charts and visualisations as it makes data and patterns easier to read\n\n\n\n\nReporting on broad ethnic minorities catageories (e.g. BAME)\n\nPublished in March 2021, the report of the Commission on Race and Ethnic Disparities made a number of suggestions on how we discuss and report on race in the UK.\nOne of the major elements was around aggregation of ethnic minorities into a single over-arching group (in particular BAME):\n\nRecommendation 24: Disaggregate the term ‘BAME’ Stop using aggregated and unhelpful terms such as ‘BAME’, to better focus on understanding disparities and outcomes for specific ethnic groups.\n\nThe reasons for this approach are two-fold:\n\nGrouping separate ethnic minorities into a single over-arching group can mask differing trends that may be present in the underlying data for groups experiencing different pressures and environments within UK society.\nThe phrasing ‘Black and Minority Ethnic’ and ‘Black, Asian and Minority Ethnic’ give an emphasis to specific ethnic groups whilst excluding others. This could be misleading or diminish consideration of the unnamed groups within the statistics.\n\nBased on this guidance, the following is how teams should aggregate and write about ethnic minority groups.\nStatistics producers should avoid where possible the practice of reporting aggregates grouping together disparate ethnic groups. Instead, statistics should be reported individually for the recommended top-level groups:\n\nWhite\nMixed / Multiple ethnic groups\nAsian / Asian British\nBlack / African / Caribbean / Black British\nOther ethnic group\n\nWhere there is a reasonable need to publish or report statistics for a full aggregation of ethnic minorities (such as reporting on existing historical targets), official guidance is as follows:\n\nUse ‘ethnic minorities’ to refer to all ethnic groups except the white British group. This term includes white minorities, such as Gypsy, Roma and Irish Traveller groups. For comparisons with the white group as a whole, use ‘ethnic minorities (excluding White minorities)’."
},
{
- "objectID": "statistics-production/embedded-charts.html#whats-in-the-template",
- "href": "statistics-production/embedded-charts.html#whats-in-the-template",
- "title": "Embedded visualisations in EES",
- "section": "What’s in the template",
- "text": "What’s in the template\nThe template provides code for some basic interactive plots. Each example plot is contained with one of the existing branches below for demonstration purposes:\n\ndemo-interactive-bar\ndemo-interactive-line"
+ "objectID": "learning-development/r.html",
+ "href": "learning-development/r.html",
+ "title": "R",
+ "section": "",
+ "text": "Guidance and tips for using the programming language R"
},
{
- "objectID": "statistics-production/embedded-charts.html#working-with-data",
- "href": "statistics-production/embedded-charts.html#working-with-data",
- "title": "Embedded visualisations in EES",
- "section": "Working with data",
- "text": "Working with data\nAs with the full dashboards, the embedded charts currently require the underlying data to be either included within the app repository on GitHub or uploaded elsewhere publicly accessible such as Google Drive or Dropbox. This currently means that any embedded charts being developed will need to use either dummy data or previously published data until the moment of publication. At the point of the parent release going live, the chart can then be updated with the latest data. Do not upload unpublished data to GitHub, Google Drive or Dropbox.\nAs described earlier, where you need to use unpublished data in your chart prior to publication, you can either a) run the chart locally in R-Studio (without pushing the unpublished data to GitHub) or b) create a DevOps/rsconnect deploy of your app, which can be temporarily used as the embed block URL. Note, this will need updating to a URL to the public dashboard on ShinyApps ready for publication.\nWe are currently developing a route to allow charts via R-Shiny apps to be hosted on DfE servers, such that draft publications will be able to incorporate embedded charts with the unpublished data. The data itself will then be accessed either from a SQL database on DfE servers."
+ "objectID": "learning-development/r.html#what-is-r",
+ "href": "learning-development/r.html#what-is-r",
+ "title": "R",
+ "section": "What is R",
+ "text": "What is R\nR is an open-source programming language specifically aimed at statisticians and data analysts."
},
{
- "objectID": "statistics-production/embedded-charts.html#specific-design-recommendations-for-embedded-plots",
- "href": "statistics-production/embedded-charts.html#specific-design-recommendations-for-embedded-plots",
- "title": "Embedded visualisations in EES",
- "section": "Specific design recommendations for embedded plots",
- "text": "Specific design recommendations for embedded plots\nExample code for producing an embeddable shiny chart is given in the template tiny shiny app repository. The following recommendations should be followed in adapting this code:\n\nFigures should be produced using plotly/ggplot2\nFigure lengths and heights should be in the range 6-10cm\nText sizes in plots should be no smaller than 12pt\nPlotly overlays should be turned off\n\nExample code for creating charts using ggplot can be found in the Using Explore Education Statistics guidance"
+ "objectID": "learning-development/r.html#what-is-r-for",
+ "href": "learning-development/r.html#what-is-r-for",
+ "title": "R",
+ "section": "What is R for",
+ "text": "What is R for\nR can be used for almost anything you can think of, notably data analysis, data visualisation, and creating reports and dashboards. It can also be used to extract data from SQL databases and run SQL queries."
},
{
- "objectID": "RAP/rap-support.html",
- "href": "RAP/rap-support.html",
- "title": "RAP support",
- "section": "",
- "text": "This page outlines the support available for RAP in DfE"
+ "objectID": "learning-development/r.html#how-to-install-r",
+ "href": "learning-development/r.html#how-to-install-r",
+ "title": "R",
+ "section": "How to install R",
+ "text": "How to install R\nDownload R (language) and RStudio (IDE) from the DfE software center. We also recommend that you download RTools (a helpful R extension) at the same time.\nThere are usually a couple of different versions available for software on the software center, we’d recommend you always go for the latest (newest) version possible."
},
{
- "objectID": "RAP/rap-support.html#email-support",
- "href": "RAP/rap-support.html#email-support",
- "title": "RAP support",
- "section": "Email support",
- "text": "Email support\n\nstatistics.development@education.gov.uk\n\n9-5, aim to reply between 1-2 days."
+ "objectID": "learning-development/r.html#best-places-to-start",
+ "href": "learning-development/r.html#best-places-to-start",
+ "title": "R",
+ "section": "Best places to start",
+ "text": "Best places to start\n\nThe DfE Analytics Academy host an online R training course. This is a great resource full of reproducible examples using DfE data. The course takes you through initially getting R downloaded, all the way through to developing apps in RShiny.\nThere is also the DfE R training guide, which is a great starting point and reference to guide you through how to get started using R and RStudio.\nAs an alternative, with a number of options for beginners to R, RStudio Education provide a variety of materials to suit different learning styles."
},
{
- "objectID": "RAP/rap-support.html#technical-workshops",
- "href": "RAP/rap-support.html#technical-workshops",
- "title": "RAP support",
- "section": "Technical workshops",
- "text": "Technical workshops\n\nIn person workshops covering specific technical skills in practice\n3 hours long, with people working in small groups\nRecently circled the sites running an introduction to Git, and introduction to R and RAP.\nContact statistics.development@education.gov.uk to register interest in workshops happening at your site or to request new topics!\n\nWe’re also considering a dedicated G6 / G7 programme to build confidence and set expectations. This may include:\n\nA 2-hour workshop explaining what teams are expected to do when creating and publishing statistics. Including tools and skills required, RAP, using EES, content design, release clearance, UE + ongoing publishing and where teams can go for help.\nThere will be take away materials that cover lines to take and where to get support.\nAll G6s and G7s are expected to do this, with repeats running to ensure everyone gets a chance to attend.\n\nKeep an eye out on teams for this programme being advertised and contact statistics.development@education.gov.uk."
+ "objectID": "learning-development/r.html#best-practice",
+ "href": "learning-development/r.html#best-practice",
+ "title": "R",
+ "section": "Best practice",
+ "text": "Best practice\nTips for reaching best practice in R can be found on our RAP page, with guidance on meeting best practice in RAP for clean final code. This makes it easier to read and pick up if another person is running your code."
},
{
- "objectID": "RAP/rap-faq.html",
- "href": "RAP/rap-faq.html",
- "title": "RAP FAQs",
- "section": "",
- "text": "This guidance is to help answer questions about RAP and act as a tool for anyone managing those implementing RAP."
+ "objectID": "learning-development/r.html#how-to-work-with-r",
+ "href": "learning-development/r.html#how-to-work-with-r",
+ "title": "R",
+ "section": "How to work with R",
+ "text": "How to work with R\n\nR Projects\n\nWhenever you are using R, you should work in an RProject. This just makes sure you are set up in the correct working directory, so your code is pointing at the right folders and files.\nThis guide for using projects in R is a really useful article to help you set up a project.\nYou can check which project you are working in by looking in the top right hand corner of RStudio:\n\n\n\n\nOutlines\n\nIn RStudio you can greatly increase the navigability of your code by taking advantage of outlines. More information on folding and navigating outlines in RStudio can be found online, though when using rmarkdown reports, remember to use names first, such as ## Rows that aren't matching: r nrow(joined %>% filter(matching == FALSE)), rather than having the R code first, so that they are easy to discern in the outline.\n\n\n\nrenv\n\nYou should use the renv package for package and version control in R.\nPackages and versions of R regularly update. Over time, this can cause code to break - e.g. if different dependencies are required for later versions of packages to work. Using renv creates a “snapshot” of your code and packages at the time you created it, which anyone can then recreate when they come to use your code.\nThis is really important for reproducibility, and will help you meet elements of great practice with recyclable code for future use.\n\n\nrenv::restore()\n\nSometimes renv::restore() can fail, and when in specific renv-controlled projects install.packages() will fail saying that packages aren’t available even when they clearly are. There are a couple of workarounds we have found that get around this failure.\n\nConfiguring the proxy settings by running the below in R - this also helps if you are getting timeout issues when trying to webscrape with R:\n\nSys.setenv(no_proxy=\"*\") \n\n\nSpecifying the renv library as the install location. It’s a bit of a fudge, though these lines are helpful to get the packages from the renv lockfile installed and you running the project when needed:\n\nmyPath <- .libPaths()[1]\n\nforceInstall <- function(pkg, path) {\nmissing <- suppressWarnings(eval(parse(text= paste0(\"!require(\",pkg,\")\"))))\n\nif(missing == FALSE){\nmessage(pkg, \" is already installed.\")\n} else{\ninstall.packages(pkg, lib = path)\n}\n}\n\nforceInstall(\"jsonlite\", myPath)\n\nrenvPackages <- names(jsonlite::fromJSON(\"renv.lock\", flatten = TRUE)$Packages)\n\ninvisible(lapply(renvPackages, forceInstall, path = myPath))\nMore manual equivalent to use for specific packages:\n.libPaths() # note down output 1, and reuse in the lib argument of install.packages() as below\n\ninstall.packages(\"rmarkdown\", lib = \"C:/Users/swong/OneDrive - Department for Education/Documents/stats-production-guidance/renv/library/R-4.0/x86_64-w64-mingw32\")\n\n\n\n\nUpdating packages in renv\n\nTo update a single package run:\nrenv::update(\"dplyr\")\nTo update all packages run:\nrenv::update()\n\n\n\nInstalling old package versions in renv\n\nThis is surprisingly neat to do. Let’s say you wanted to roll back to version 1.0.2 of dplyr, you would run the following:\nrenv::install(\"dplyr@1.0.2\")"
},
{
- "objectID": "RAP/rap-faq.html#what-is-rap",
- "href": "RAP/rap-faq.html#what-is-rap",
- "title": "RAP FAQs",
- "section": "What is RAP",
- "text": "What is RAP\nGeneral questions about Reproducible Analytical Pipelines and why analysts need to care about it.\n\n\nWhat is RAP, is it just using R?\n\nNo. Reproducible Analytical Pipelines (RAPs) are automated statistical and analytical processes. They incorporate elements of software engineering best practice to ensure that the pipelines are reproducible, auditable, efficient, and high quality.\nDoing RAP is doing analysis using the best methods available to us, which is an expectation of the statistics code of practice.\nThe tools we recommend for statistics production RAP are SQL, Git and R. Other suitable alternatives that allow you to meet the core principles can also be used, but you should check this with the Statistics Development Team first. Ideally any tools used would be open source, Python is a good example of a tool that would also be well suited, though is less widely used in DfE and has a steeper learning curve than R.\nUsing R for parts of the process does get you close to a lot of the RAP principles with little effort, which is why we recommend it as one of the tools you should use.\nMore details and learning resources for the recommended tools can be found in our appropriate tools guidance.\n\n\n\nIs this specific to DfE?\n\nNo – RAP is a cross government strategy, and all departments are expected to use this way of working. See the analytical function page on RAP to see other examples of departments utilising RAP and seeing the benefits.\nHere is a great blog post from NHS digital on ‘Why we’re getting our data teams to RAP’.\nRAP is also a strategic objective of Analysis function strategy for 22-25:\n\ndelivering the Reproducible Analytical Pipelines (RAP) Strategy and action plan to embed RAP across government, ensuring our processes are automated, efficient, and high quality which frees up resource to add value and insight in our analysis.\n\n\n\n\nI’m overwhelmed by all the steps, is RAP really necessary?\n\nThe levels within RAP good practice are in line with what a lot of teams are doing already, take time to understand what the different steps mean and don’t panic about the quantity of them as they’ve intentionally been broken down into the smallest level that was sensible.\nStatistics publications are some of the most important pieces of statistics work that the department does. It’s important we get that data right; RAP can help us automate the production and QA of outputs in a way that gives us the most confidence in the statistics and causes the least burden for you.\n\n\n\nWill implementing RAP lead to a disconnect with the data and ‘black box’ processes?\n\nNot at all. When fully implemented and accompanied with the relevant skills, RAP has the opposite effect as each stage of the process is clearly written as code.\nWe recognise there is a sizeable skill gap, and that until this is addressed there can be a risk of feeling like code is a black box. This isn’t a reason to avoid RAP though, instead it’s a big reason why we need to push ahead with prioritising upskilling ourselves to implement it.\n\n\n\nTeams vary, and what my team does is different to others, if I’m happy with my approach, can I ignore some of the RAP steps?\n\nNo, you should not ignore any of the steps. If you think any of the steps aren’t applicable to you, talk to the Statistics Development Team so we can understand more about the processes in your area. It may well be that you’re meeting it without realising, there’s a misunderstanding, or there’s something more we can add to the guidance.\nIf you have unique or nuanced processes, RAP helps you document these and make your area more resilient to turnover of people. RAP is all about automating the process and documenting it in a standardised way to make it easily reproducible."
+ "objectID": "learning-development/r.html#quick-reference-lookup",
+ "href": "learning-development/r.html#quick-reference-lookup",
+ "title": "R",
+ "section": "Quick reference lookup",
+ "text": "Quick reference lookup\n\nIf you want a useful guide for R syntax or functions, then look no further than the R cheat sheets, these can be an invaluable point of reference. Below we’ve included a few particularly relevant ones:\n\nIntroduction to the RStudio environment\nBase R\nImporting data into R\ndplyr for data manipulation\nstringr for string manipulation\nRegex\nRMarkdown\nRShiny\nggplot2 for data visualisations\npurrr for applying functions"
},
{
- "objectID": "RAP/rap-faq.html#getting-started",
- "href": "RAP/rap-faq.html#getting-started",
- "title": "RAP FAQs",
- "section": "Getting started",
- "text": "Getting started\nQuestions about how to get started with RAP.\n\n\nCan I leave the R stuff to others in my team whilst I focus elsewhere?\n\nNo. Anyone undertaking any analytical processes should now be using RAP processes, and coding is a critical analytical skill. RAP is a cross-government strategy and there is an expectation in all departments for all analysts to move to this way of working.\nWe all must ensure that analysis is reproducible, transparent, and robust using coding and code management best practices (GSG competency framework).\n\n\n\nImplementing RAP takes time to setup initially, how can we prioritise it?\n\nThere is a clear expectation that this is the direction for analysis in government, ultimately, it’s a part of the new digital skills required for the role of analysts. If we’re not prioritising improving our analysis in line with RAP principles, we’re not approaching our work correctly.\nWe have support from senior leadership, and this is a department priority, so you should be building in time for it. If you are having difficulties prioritising RAP after talking with your line management chain, please get in touch with the Statistics Development Team.\nIn the long term, implementing RAP will significantly reduce the time it takes to run your pipeline, and so while it requires time upfront, it creates more time in the future to prioritise other things. It also improves the quality and reproducibility of our work, giving numerous business benefits."
+ "objectID": "learning-development/r.html#other-resources",
+ "href": "learning-development/r.html#other-resources",
+ "title": "R",
+ "section": "Other resources",
+ "text": "Other resources\n\nHere is another free introduction to R course by Quantargo.\nR Markdown: The Definitive Guide, hopefully this one should be relatively self-explanatory!\nData science in education provides a heavily detailed guide for beginners in R learning to process data, with some well written out sections that may be of interest.\nHandy guide to collapsing and sectioning R code for easy navigation in RStudio.\nHere are 5 handy tidyverse functions that you should know if you’re using R to process data. Number two is especially useful for those processing wide data into a tidy format!\nMoJ have produced guidance on writing functions in R\nIf you’re wondering how best to make the jump to R from Excel and SQL, take a look at this coffee and coding presention by David Sands.\nMalcolm Barrett has done some slides on dplyr, ggplot2, and using purrr which may be useful if you’re looking at learning more about any of those packages.\nAlso check out the janitor package, it has some particularly powerful functions that are worth a look for tidying and QA’ing data."
},
{
- "objectID": "RAP/rap-faq.html#tools-for-rap",
- "href": "RAP/rap-faq.html#tools-for-rap",
- "title": "RAP FAQs",
- "section": "Tools for RAP",
- "text": "Tools for RAP\nQuestions about what tools and software to use when applying RAP principles.\n\n\nDo I need to rewrite existing SQL code into R?\n\nNo. In fact, we recommend you don’t necessarily do this!\nSQL is a useful tool in its own right and some processes are better using SQL code executed on the databases.\nWhat we recommend, is that all of your code is within a single R project, and the SQL scripts are executed via R, this helps with documenting the process, but keeps the SQL code and benefits of doing heavy processes on the database side. See the final question in this ‘Tools for RAP’ section for more information.\n\n\n\nI’m sometimes limited by the access and tools I have (ESFA servers, EDAP, Databricks), is there anything that can be done about this?\n\nThe first step is to let us (the Statistics Development Team) know so we can understand the wider landscape and escalate. There isn’t really a quick fix, but the first step is raising awareness.\n\n\n\nWhat happens if we can’t reproduce our current processes using R?\n\nThis is highly unlikely and is more likely to be from a lack of knowledge of what is possible in R – if you’re struggling to reproduce processes, please contact the Statistics Development Team so they’re aware of your processes and can help you implement RAP principles.\n\n\n\nR isn’t always the quickest tool for processing data, can we use SQL instead?\n\nYes. We recommend different tools for different purposes, SQL should be used for querying source data and large data processes, R should be used for more complicated data manipulation, QA, and analysis. There’s a grey area in the middle where you can do some things in either tool, sometimes you’ll need to test for yourself which way is faster (e.g., linking datasets or transforming tables).\nSQL scripts should be run in an automated way using R, instead of manually running them and copying the outputs. The difference we’re talking about here is whether you process data on the database server or on the machine you’re running R. A simplistic rule of thumb is do large scale processing on the database side (e.g., filtering and aggregating), and then only bring the data you need for manipulation / complicated processing into R."
+ "objectID": "learning-development/r.html#excel-functions-in-r",
+ "href": "learning-development/r.html#excel-functions-in-r",
+ "title": "R",
+ "section": "Excel functions in R",
+ "text": "Excel functions in R\nR can do everything you do in excel, but takes out the human error. The reference table below shows how you would carry out popular Excel commands in R.\nR comes in with a built-in dataset called “iris”. We’ll use this for all examples so you can recreate them in your local area.\nREMEMBER: R is case sensitive, so all references to column names/entries need to be as-is in the dataset you are looking at. Functions exist that can translate all your columns to lower or snake case for ease!\n\n\n\n\n\n\n\n\nCommon Excel Task\nExample with iris \nHow to do in R with dplyr\n\n\n\n\nSelect specific columns\nSelect only species and petal length\niris %>% select(Species, Petal.Length)\n\n\nList unique entries in field (column)\nFind the unique entries for the “Species” column in iris\niris %>% select(Species) %>% distinct()\n\n\nFilter/select based on criteria\nFilter for sepal length >4 and sepal width <2.5, but NOT “versicolor” species\niris %>% filter(Sepal.Length > 4 & Sepal.Width <2.5 & Species != \"versicolor\")\n\n\nFilter for multiple criteria in same column\nFilter for all “setosa” and “versicolor” species\niris %>% filter(Species %in% c(\"setosa\", \"versicolor\")\n\n\nIf else with OR\nCreate new column called “size_group” based on length or width of petal\niris %>% mutate(size_group =if_else( Petal.Length > 4 | Petal.Width >1.5, \"Large\", \"Small\"))\n\n\nMultiple if else\nCreate new column called “flower_price” based on species and petal length\niris %>% mutate(flower_price = case_when( Species == \"setosa\" & Petal.Length > 1.5 ~\"top band\",Species == \"versicolor\" & Petal.Length < 4 ~\"low_band\", TRUE ~ \"mid_band\"))\n\n\nCOUNTIF\nCount the number of species if they have a petal length >1.5\niris %>% filter(Petal.Length > 1.5 ) %>%group_by(Species) %>% count()\n\n\nSUMIF\nSum petal width of species if sepal width <3\niris %>% filter(Sepal.Width <3) %>%group_by(Species) %>%summarise(Petal.Width = sum(Petal.Width))\n\n\nVLOOKUP\nLookup to a table called “lookup”\niris %>% left_join(lookup, by.x=\"Species\", by.y =\"plant_species\")\n\n\nOrder by\nOrder dataset by descending petal width\niris %>% arrange(desc(Petal.Width))\n\n\n\nMore tips for moving from using Excel to using R can be found in the excel-to-R wiki."
},
{
- "objectID": "RAP/rap-faq.html#implementing-rap",
- "href": "RAP/rap-faq.html#implementing-rap",
- "title": "RAP FAQs",
- "section": "Implementing RAP",
- "text": "Implementing RAP\nQuestions on how to implement RAP.\n\n\nHaving a single script for code doesn’t seem the best way to do it, why are you suggesting this?\n\nThis is a misconception of our guidance that we will be clarifying and improving the phrasing of. The two steps this refers to are:\n\nCan the publication be reproduced using a single code script?\nAre data files produced via single code script(s) with integrated QA?’\n\nWe’re not suggesting all code lives in a single script. These steps in the RAP guidance are to encourage the repository being structured so that one clicks of a button can run the entire process from the source data through to the final output in order.\nThis means that you should have one ‘centralized’ script that details all of the different scripts in your repository. This single run script then provides a single reference point for where all of your code and scripts live, and in what order they’re executed, which is fantastic piece documentation to have!\n\n\n\nHow do I dual run for QA if our process is in code?\n\nYou don’t need to be dual running if your process is automated, it’s not the best way to QA. See our guidance on QA for more details on how you can approach it or talk to us if you’re unsure.\n\n\n\nShould I have separate repositories or branches in Git for each release?\n\nMore information on recommended ways of working with Git can be found in our guidance on repositories.\nIn short, a single repository should be used for all releases of your publication, there’s no need to have multiple as all the history is saved within previous commits.\n\n\n\nWe should have plain text documentation to accompany the process, code and comments don’t feel like enough?\n\nYes, you should have some plain text documentation, this is a part of the guidance on RAP and a part of the ‘documentation’ step to ‘good’ practice.\nThe README in your repository is the place for traditional ‘desk notes’ and text explanations of bits of the process and context required, we have guidance on writing READMEs on the statistics production website."
+ "objectID": "learning-development/r.html#sql-functions-in-r",
+ "href": "learning-development/r.html#sql-functions-in-r",
+ "title": "R",
+ "section": "SQL functions in R",
+ "text": "SQL functions in R\nR can do a lot of the things that are possible in SQL. The reference table below shows how you would carry out some popular SQL commands in R.\nREMEMBER: R is case sensitive, so all references to column names/entries need to be as-is in the dataset you are looking at. Functions exist that can translate all your columns to lower or snake case for ease!\n\n\n\n\n\n\n\nCommon SQL Task\nHow to do in R (with dplyr)\n\n\n\n\nSELECT * FROM TABLE\ntable %>% select()\n\n\nSELECT ColA, ColB, ColC FROM TABLE\ntable %>% select(ColA, ColB, ColC)\n\n\nSELECT DISTINCT ColA FROM TABLE\ntable %>% select(ColA) %>% distinct()\n\n\nTABLE A LEFT JOIN (TABLE B) ON TABLEA.x = TABLEB.y\ntableA %>% left_join(TableB, by = c(x = y))\n\n\nCASE WHEN x = 1 THEN 1, WHEN x =2 THEN 2, ELSE 0 END AS New_column_name\n%>% mutate (New_column_name = case_when (x == 1 ~ 1, x == 2 ~ 2, TRUE ~ 0))\n\n\nCONCAT(LEA, ESTAB) AS LAESTAB\n%>% mutate(LAESTAB = paste0(LEA, ESTAB))\n\n\nSELECT COUNT(*) FROM TABLE\ntable %>% nrow()\n\n\nSELECT COUNT(ColA) FROM TABLE\ntable %>% count(colA)\n\n\nSELECT Date_column = CONVERT(DATE, Date_column) FROM TABLE\ntable %>% mutate(Date_column = as.Date(Date_column))\n\n\nSELECT Number_column = CONVERT(INT, Number_column ) FROM TABLE\ntable %>% mutate(Number_column = as.numeric(Number_column))\n\n\nSELECT String_column = CONVERT(VARCHAR, String_column ) FROM TABLE\ntable %>% mutate(String_column = as.character(String_column))\n\n\nDROP TableA\nrm(TableA)\n\n\n\nMore tips for moving from using SQL to using R can be found in the SQL-to-R wiki."
},
{
- "objectID": "RAP/rap-faq.html#learning-about-rap",
- "href": "RAP/rap-faq.html#learning-about-rap",
- "title": "RAP FAQs",
- "section": "Learning about RAP",
- "text": "Learning about RAP\nQuestions learning more about RAP and developing skills.\n\n\nCan I look at what other people are doing?\n\nAbsolutely. Currently the best way to do this is to ask other teams to share code directly with you. If we can all make more progress with using Git, then this will make it much easier to share repositories and have code open by default for interested analysts to browse.\nWe are also starting up a knowledge share series, so keep an eye out for that and make sure to contribute yourselves. We’d also encourage analysts to make more use of teams to post questions and ask about other people in our area doing similar types of analysis, no question is too big or too small!\n\n\n\nI don’t have the skills to implement RAP, how do I get them?\n\nSee the top section on the support on offer!\nPlus, if you’re ever unsure at all, you can always contact statisics.development@education.gov.uk who will be able to help you find resources that work for you or do bespoke training for your team’s needs."
+ "objectID": "learning-development/r.html#tips-for-using-r",
+ "href": "learning-development/r.html#tips-for-using-r",
+ "title": "R",
+ "section": "Tips for using R",
+ "text": "Tips for using R\nA selection of handy bits of code and workarounds for common issues. More useful code snippets can also be found in our github repo\n\n\nSpecifying a version of R to use\n\nThis can be done most easily by navigating in RStudio through Tools > Global options > General > Basic > R version (change). It’s likely you’ll need to restart RStudio for the changes to take affect.\n\n\n\nRounding\n\nThe base R function of round() rounds 5’s downwards. To round them upwards you can create a custom function like the one below:\nroundFiveUp <- function(x, n){ \n z = abs(x)*10^n \n z = z + 0.5 + sqrt(.Machine$double.eps) \n z = trunc(z) \n z = z/10^n \n positiveNegative = sign(x) \n return(z * positiveNegative) \n}\nThis function can also be found in the dfeR R package.\n\n\n\nPassing variables as arguments\n\nThis can be worked around by using a combination of eval() and parse(), as shown in the below function:\nshowFilterLevels <- function(data, meta) {\n filters <- meta %>%\n filter(col_type == \"Filter\") %>%\n pull(col_name)\n\n levelsTable <- function(filter) {\n return(eval(parse(text = paste0(\"data %>% select(\", filter, \") %>% distinct()\"))))\n }\n\n output <- lapply(filters, levelsTable)\n\n return(output)\n}\n\n\n\nReverse additive filters\n\nYou might want to filter your dataset based on multiple negative conditions. Normally to filter on multiple conditions, you would use filter(condition1 & condition2). The “filter” function does not work well with negative conditions (i.e. filtering for cases where condition 1 and condition 2 are not met). Instead, you can use subset(!(condition1 & condition2).\n\n\n\nFile locations\n\nStruggling to get files to talk to one another, or get code to find and use another R script? Use here::here() and marvel at it’s wondrous ability to magic away issues.\n\n\n\nInterweaving vectors\n\nThere’s an easy way to interweave multiple vectors into one single vector using c(rbind()). The example below shows two vectors, but you can have even more if you need.\n#Two vectors, x and y\nx <- 1:3\ny <- 4:6\n\n#Run code to interweave\nc(rbind(x, y))\n\n#Output below\n# [1] 1 4 2 5 3 6\n\n\n\nMaking charts interactive\n\nWhen pulling ggplot charts into RMarkdown reports, you can consider making them even more user-friendly and interactive with plotly. Further information on how to make your charts interactive with plotly can be found online.\n#Simple ggplot chart called \"p\"\np <- ggplot(dat, aes(x=xvar, y=yvar)) +\n geom_point(shape=1) # Use hollow circles\n\n#Apply ggplotly() to it to make it interactive\nfig <- ggplotly(p)\n\n\n\n\nReplace all values with another\n\nHave you ever needed to replace every value in your data with another? This can come in handy when you are looking at suppression, e.g. converting all NAs to “z” or all values under a certain threshold to “c”.\ndata %>% mutate_all(~ replace(., . == \"Value to replace\", \"Replacement\"))\n\n\n\n\nTemporary groups\n\nThe group_by() function in dplyr is really useful, but can be fiddly if you only want to use it for one operation in a chunk of code. The with_groups() function from dplyr lets you do this, saving you having to group and ungroup data each time.\ndata %>% mutate(annual_average = with_groups(time_period, mean))\n\n\n\n\nFinding package dependencies\n\nOften we’ll take chunks of code and reuse them for new projects. This can lead to building up a long list of packages to install, not all of which end up being used in your new code. The NCmisc package is a really handy way to check which packages and functions are used in your code.\nFirstly, load up all the packages the code has library() commands for, then run the following:\nlist.functions.in.file('your-filename-here.R', alphabetic = TRUE)\n\n\n\nVisualise dependencies\n\nThe depgraph package allows you to plot a graph of all the dependencies in your R project, which can be a useful tool to help you cut down on the number of package dependencies. Briefly, in these graphs you can look for “hot spots” in the network (big bright dots), which represent packages that have many upstream dependencies but are potentially easy to remove because they have few downstream dependencies (that is, only your package depends on them).\nplot_dependency_graph(\n pkg = multibridge_pkg\n , suggests = FALSE\n , option = \"cividis\"\n)\n\n\n\n\nReproducible random numbers\n\nThe set.seed() function generates a sequence of random numbers, starting from the value you define in the brackets. This ensures you get the same sequence of random numbers each time you run set.seed() with the same value, which is helpful to test that your results are reproducible.\n# random sampling\n> sample(LETTERS, 5)\n[1] \"K\" \"N\" \"R\" \"Z\" \"G\"\n> sample(LETTERS, 5)\n[1] \"L\" \"P\" \"J\" \"E\" \"D\"\n\n# reproducible random sampling\n> set.seed(42); sample(LETTERS, 5)\n[1] \"Q\" \"E\" \"A\" \"J\" \"D\"\n> set.seed(42); sample(LETTERS, 5)\n[1] \"Q\" \"E\" \"A\" \"J\" \"D\"\n\n\n\n\nAutomatic logging\n\nThe tidylog package is a really useful tool for providing automated feedback on dplyr and tidyr operations.\nlibrary(tidylog)\n\nfiltered <- filter(mtcars, cyl == 4)\n#> filter: removed 21 rows (66%), 11 rows remaining\nmutated <- mutate(mtcars, new_var = wt ** 2)\n#> mutate: new variable 'new_var' (double) with 29 unique values and 0% NA\n\n\n\nRunning SQL scripts from R\n\nR can be used to execute SQL scripts to extract data from a database as well as querying the database directly via R. For using R to execute a SQL script you’ll need the SQL script/s to be in your R Project and to make a connection via R to the database.\n# Library calls ====\n\nlibrary(odbc)\nlibrary(DBI)\n\n# DB connection ====\n\ncon <- DBI::dbConnect(odbc::odbc(),\n Driver = \"ODBC Driver 17 for SQL Server\",\n Server = \"server_name\",\n Database = \"database_name\",\n UID = \"\",\n PWD = \"\",\n Trusted_Connection = \"Yes\"\n)\n\n# Function to read in sql scripts ====\n\ngetSQL <- function(filepath){\n con = file(filepath, \"r\")\n sql.string <- \"\"\n \n while (TRUE){\n line <- readLines(con, n = 1)\n \n if ( length(line) == 0 ){\n break\n }\n \n line <- gsub(\"\\\\t\", \" \", line)\n \n if(grepl(\"--\",line) == TRUE){\n line <- paste(sub(\"--\",\"/*\",line),\"*/\")\n }\n \n sql.string <- paste(sql.string, line)\n }\n \n close(con)\n return(sql.string)\n}\n\n# Execute SQL query and pull into R ====\n\nmy_table <- dbGetQuery(con, getSQL(\"my_query_script.sql\"))\n\nIf you’re struggling to get your SQL scripts to execute from R, try adding the following two lines to the top of your SQL script to force the formatting required for it to work:\nSET ANSI_PADDING OFF\nSET NOCOUNT ON;\n\n\n\nCan’t find make error\n\nThis error will appear when trying to restore old package versions using renv::restore(). It is usually due to Rtools not being properly installed, and therefore your system is struggling to install the older version of the package that hasn’t been provided by CRAN.ware centre.\nThe solution takes a few minutes, but is relatively straightforward, you can install Rtools as a direct download from CRAN. On there, pick the right version of RTools for your version of R, download the file and install.\nSet the install location to your c drive, for example, if installing for R4.2.2, install RTools42 to C:\\rtools42 (this is the default location).\nOnce it’s installed close and reopen RStudio and run renv::restore() again - it should hopefully now restore and install your packages successfully!\n\n\n\nPackages not downloading due to “wininet” error (renv 0.17)\n\nPackages are generally downloaded from CRAN using one of two methods: wininet or curl. Since renv version 0.17, analysts potentially hit an error associated with renv defaulting to wininet, whilst R defaults to curl, causing package downloads to fail. To fix this issue, analysts should change their default download method with renv to be curl. This can be done across all R-projects on your machine, by setting the default in your global R environment file. To do this, open up a bash terminal and enter the following command:\necho 'RENV_DOWNLOAD_METHOD = \"curl\")' >> ~/.Renviron\nThen restart R-Studio and downloads with renv should now succeed."
},
{
"objectID": "learning-development/sql.html",
@@ -532,312 +497,382 @@
"text": "Best places to start\nThe Python Software Foundation provides an introduction for beginners to Python.\nWe recommend making use of the following communities: - Python DfE community on teams"
},
{
- "objectID": "learning-development/git.html",
- "href": "learning-development/git.html",
- "title": "Git",
+ "objectID": "RAP/rap-faq.html",
+ "href": "RAP/rap-faq.html",
+ "title": "RAP FAQs",
"section": "",
- "text": "Guidance and tips for version control with Git"
+ "text": "This guidance is to help answer questions about RAP and act as a tool for anyone managing those implementing RAP."
},
{
- "objectID": "learning-development/git.html#what-is-git",
- "href": "learning-development/git.html#what-is-git",
- "title": "Git",
- "section": "What is Git",
- "text": "What is Git\nIt is a version control software. It is by far the best of its kind and is widely used by software developers and data scientists."
+ "objectID": "RAP/rap-faq.html#what-is-rap",
+ "href": "RAP/rap-faq.html#what-is-rap",
+ "title": "RAP FAQs",
+ "section": "What is RAP",
+ "text": "What is RAP\nGeneral questions about Reproducible Analytical Pipelines and why analysts need to care about it.\n\n\nWhat is RAP, is it just using R?\n\nNo. Reproducible Analytical Pipelines (RAPs) are automated statistical and analytical processes. They incorporate elements of software engineering best practice to ensure that the pipelines are reproducible, auditable, efficient, and high quality.\nDoing RAP is doing analysis using the best methods available to us, which is an expectation of the statistics code of practice.\nThe tools we recommend for statistics production RAP are SQL, Git and R. Other suitable alternatives that allow you to meet the core principles can also be used, but you should check this with the Statistics Development Team first. Ideally any tools used would be open source, Python is a good example of a tool that would also be well suited, though is less widely used in DfE and has a steeper learning curve than R.\nUsing R for parts of the process does get you close to a lot of the RAP principles with little effort, which is why we recommend it as one of the tools you should use.\nMore details and learning resources for the recommended tools can be found in our appropriate tools guidance.\n\n\n\nIs this specific to DfE?\n\nNo – RAP is a cross government strategy, and all departments are expected to use this way of working. See the analytical function page on RAP to see other examples of departments utilising RAP and seeing the benefits.\nHere is a great blog post from NHS digital on ‘Why we’re getting our data teams to RAP’.\nRAP is also a strategic objective of Analysis function strategy for 22-25:\n\ndelivering the Reproducible Analytical Pipelines (RAP) Strategy and action plan to embed RAP across government, ensuring our processes are automated, efficient, and high quality which frees up resource to add value and insight in our analysis.\n\n\n\n\nI’m overwhelmed by all the steps, is RAP really necessary?\n\nThe levels within RAP good practice are in line with what a lot of teams are doing already, take time to understand what the different steps mean and don’t panic about the quantity of them as they’ve intentionally been broken down into the smallest level that was sensible.\nStatistics publications are some of the most important pieces of statistics work that the department does. It’s important we get that data right; RAP can help us automate the production and QA of outputs in a way that gives us the most confidence in the statistics and causes the least burden for you.\n\n\n\nWill implementing RAP lead to a disconnect with the data and ‘black box’ processes?\n\nNot at all. When fully implemented and accompanied with the relevant skills, RAP has the opposite effect as each stage of the process is clearly written as code.\nWe recognise there is a sizeable skill gap, and that until this is addressed there can be a risk of feeling like code is a black box. This isn’t a reason to avoid RAP though, instead it’s a big reason why we need to push ahead with prioritising upskilling ourselves to implement it.\n\n\n\nTeams vary, and what my team does is different to others, if I’m happy with my approach, can I ignore some of the RAP steps?\n\nNo, you should not ignore any of the steps. If you think any of the steps aren’t applicable to you, talk to the Statistics Development Team so we can understand more about the processes in your area. It may well be that you’re meeting it without realising, there’s a misunderstanding, or there’s something more we can add to the guidance.\nIf you have unique or nuanced processes, RAP helps you document these and make your area more resilient to turnover of people. RAP is all about automating the process and documenting it in a standardised way to make it easily reproducible."
+ },
+ {
+ "objectID": "RAP/rap-faq.html#getting-started",
+ "href": "RAP/rap-faq.html#getting-started",
+ "title": "RAP FAQs",
+ "section": "Getting started",
+ "text": "Getting started\nQuestions about how to get started with RAP.\n\n\nCan I leave the R stuff to others in my team whilst I focus elsewhere?\n\nNo. Anyone undertaking any analytical processes should now be using RAP processes, and coding is a critical analytical skill. RAP is a cross-government strategy and there is an expectation in all departments for all analysts to move to this way of working.\nWe all must ensure that analysis is reproducible, transparent, and robust using coding and code management best practices (GSG competency framework).\n\n\n\nImplementing RAP takes time to setup initially, how can we prioritise it?\n\nThere is a clear expectation that this is the direction for analysis in government, ultimately, it’s a part of the new digital skills required for the role of analysts. If we’re not prioritising improving our analysis in line with RAP principles, we’re not approaching our work correctly.\nWe have support from senior leadership, and this is a department priority, so you should be building in time for it. If you are having difficulties prioritising RAP after talking with your line management chain, please get in touch with the Statistics Development Team.\nIn the long term, implementing RAP will significantly reduce the time it takes to run your pipeline, and so while it requires time upfront, it creates more time in the future to prioritise other things. It also improves the quality and reproducibility of our work, giving numerous business benefits."
+ },
+ {
+ "objectID": "RAP/rap-faq.html#tools-for-rap",
+ "href": "RAP/rap-faq.html#tools-for-rap",
+ "title": "RAP FAQs",
+ "section": "Tools for RAP",
+ "text": "Tools for RAP\nQuestions about what tools and software to use when applying RAP principles.\n\n\nDo I need to rewrite existing SQL code into R?\n\nNo. In fact, we recommend you don’t necessarily do this!\nSQL is a useful tool in its own right and some processes are better using SQL code executed on the databases.\nWhat we recommend, is that all of your code is within a single R project, and the SQL scripts are executed via R, this helps with documenting the process, but keeps the SQL code and benefits of doing heavy processes on the database side. See the final question in this ‘Tools for RAP’ section for more information.\n\n\n\nI’m sometimes limited by the access and tools I have (ESFA servers, EDAP, Databricks), is there anything that can be done about this?\n\nThe first step is to let us (the Statistics Development Team) know so we can understand the wider landscape and escalate. There isn’t really a quick fix, but the first step is raising awareness.\n\n\n\nWhat happens if we can’t reproduce our current processes using R?\n\nThis is highly unlikely and is more likely to be from a lack of knowledge of what is possible in R – if you’re struggling to reproduce processes, please contact the Statistics Development Team so they’re aware of your processes and can help you implement RAP principles.\n\n\n\nR isn’t always the quickest tool for processing data, can we use SQL instead?\n\nYes. We recommend different tools for different purposes, SQL should be used for querying source data and large data processes, R should be used for more complicated data manipulation, QA, and analysis. There’s a grey area in the middle where you can do some things in either tool, sometimes you’ll need to test for yourself which way is faster (e.g., linking datasets or transforming tables).\nSQL scripts should be run in an automated way using R, instead of manually running them and copying the outputs. The difference we’re talking about here is whether you process data on the database server or on the machine you’re running R. A simplistic rule of thumb is do large scale processing on the database side (e.g., filtering and aggregating), and then only bring the data you need for manipulation / complicated processing into R."
+ },
+ {
+ "objectID": "RAP/rap-faq.html#implementing-rap",
+ "href": "RAP/rap-faq.html#implementing-rap",
+ "title": "RAP FAQs",
+ "section": "Implementing RAP",
+ "text": "Implementing RAP\nQuestions on how to implement RAP.\n\n\nHaving a single script for code doesn’t seem the best way to do it, why are you suggesting this?\n\nThis is a misconception of our guidance that we will be clarifying and improving the phrasing of. The two steps this refers to are:\n\nCan the publication be reproduced using a single code script?\nAre data files produced via single code script(s) with integrated QA?’\n\nWe’re not suggesting all code lives in a single script. These steps in the RAP guidance are to encourage the repository being structured so that one clicks of a button can run the entire process from the source data through to the final output in order.\nThis means that you should have one ‘centralized’ script that details all of the different scripts in your repository. This single run script then provides a single reference point for where all of your code and scripts live, and in what order they’re executed, which is fantastic piece documentation to have!\n\n\n\nHow do I dual run for QA if our process is in code?\n\nYou don’t need to be dual running if your process is automated, it’s not the best way to QA. See our guidance on QA for more details on how you can approach it or talk to us if you’re unsure.\n\n\n\nShould I have separate repositories or branches in Git for each release?\n\nMore information on recommended ways of working with Git can be found in our guidance on repositories.\nIn short, a single repository should be used for all releases of your publication, there’s no need to have multiple as all the history is saved within previous commits.\n\n\n\nWe should have plain text documentation to accompany the process, code and comments don’t feel like enough?\n\nYes, you should have some plain text documentation, this is a part of the guidance on RAP and a part of the ‘documentation’ step to ‘good’ practice.\nThe README in your repository is the place for traditional ‘desk notes’ and text explanations of bits of the process and context required, we have guidance on writing READMEs on the statistics production website."
+ },
+ {
+ "objectID": "RAP/rap-faq.html#learning-about-rap",
+ "href": "RAP/rap-faq.html#learning-about-rap",
+ "title": "RAP FAQs",
+ "section": "Learning about RAP",
+ "text": "Learning about RAP\nQuestions learning more about RAP and developing skills.\n\n\nCan I look at what other people are doing?\n\nAbsolutely. Currently the best way to do this is to ask other teams to share code directly with you. If we can all make more progress with using Git, then this will make it much easier to share repositories and have code open by default for interested analysts to browse.\nWe are also starting up a knowledge share series, so keep an eye out for that and make sure to contribute yourselves. We’d also encourage analysts to make more use of teams to post questions and ask about other people in our area doing similar types of analysis, no question is too big or too small!\n\n\n\nI don’t have the skills to implement RAP, how do I get them?\n\nSee the top section on the support on offer!\nPlus, if you’re ever unsure at all, you can always contact statisics.development@education.gov.uk who will be able to help you find resources that work for you or do bespoke training for your team’s needs."
+ },
+ {
+ "objectID": "RAP/rap-statistics.html",
+ "href": "RAP/rap-statistics.html",
+ "title": "RAP for Statistics",
+ "section": "",
+ "text": "Guidance for how to implement the principles of Reproducible Analytical Pipelines (RAP) into statistics production processes"
+ },
+ {
+ "objectID": "RAP/rap-statistics.html#our-scope",
+ "href": "RAP/rap-statistics.html#our-scope",
+ "title": "RAP for Statistics",
+ "section": "Our scope",
+ "text": "Our scope\nWe want to focus on the parts of the production process that we have ownership and control over – so we are focussing on the process from data sources to publishable data files. This is the part of the process where RAP can currently add the most value - automating the production and quality assurance of our outputs currently takes up huge amount of analytical resource, which could be better spent providing insight and other value adding activity. \n\n\n\nIn Official Statistics production we are using RAP as a framework for best practice when producing our published data files, as these are the foundations of our publications moving forward. Following this framework will help us to improve and standardise our current production processes and provide a clear ‘pipeline’ for analysts to follow. This will have the added benefit of setting a clear and defined world of tools and skills required, making learning and development that much clearer and easier. To get started with RAP, we first need to be able to understand what it actually means in practice, and be able to assess our own work against the principles of RAP.\nImplementing RAP for us will involve combining the use of SQL, R, and clear, consistent version control to increase efficiency and accuracy in our work. For more information on what these tools are, why we are using them, and resources to help up-skill in those areas, see our learning resources page.\nThe collection of, and routine checking of data as it is coming into the department is also an area that RAP can be applied to. We have kept this out of scope at the moment as the levels of control in this area vary wildly from team to team. If you would like advice and help to automate any particular processes, feel free to contact us."
+ },
+ {
+ "objectID": "RAP/rap-statistics.html#how-to-assess-your-publication",
+ "href": "RAP/rap-statistics.html#how-to-assess-your-publication",
+ "title": "RAP for Statistics",
+ "section": "How to assess your publication",
+ "text": "How to assess your publication\nThe checklist provided in the publication self-assessment tool is designed to make reviewing our processes against our RAP levels easier, giving a straightforward list of questions to check your work against. This will flag potential areas of improvement, and you can then use the links to go to the specific section with more detail and guidance on how to develop your current processes in line with best practice.\nSome teams will already be looking at best practice, while others will still have work to do to achieve the department’s baseline of good and great practice. We know that all teams are starting this from different points, and are here to support all teams from their respective starting positions."
+ },
+ {
+ "objectID": "RAP/rap-statistics.html#where-we-need-to-focus",
+ "href": "RAP/rap-statistics.html#where-we-need-to-focus",
+ "title": "RAP for Statistics",
+ "section": "Where we need to focus",
+ "text": "Where we need to focus\nMost teams have already made progress with their production of tidy data files, and the release of the automated screener has now tied up that end point of the pipeline that we are all currently working towards. The standard pipeline for all teams will roughly resemble this:\n\n\n\nThe key now is for us to build on the work so far and focus on how we improve the quality and efficiency of our production processes up to that point. To do this, we need to make a concerted effort to standardise how we store and access our data, before then automating what we can to reduce the burden of getting the numbers ready and see the benefits of RAP. The exact meaning of this will vary within teams."
},
{
- "objectID": "learning-development/git.html#what-is-git-for",
- "href": "learning-development/git.html#what-is-git-for",
- "title": "Git",
- "section": "What is Git for",
- "text": "What is Git for\nGit is a version control software that tracks changes to files within a folder that you assign Git to track. It works best with plain text files such as flat data files, code scripts and markdown documents. These folders are known as repositories and can be held and managed securely in a central online place such as GitHub (best for public), GitLab (can be good for either public or private) and Azure DevOps (best for private). We can easily mirror our Azure DevOps repositories in the DfE Analytical Services area on GitHub.\nIt is widely used across DfE and integrates neatly with our use of Azure DevOps, as well as being the current leading version control software in the world of coding with over 87% of 74,298 stack overflow users using it."
+ "objectID": "RAP/rap-statistics.html#how-to-get-started",
+ "href": "RAP/rap-statistics.html#how-to-get-started",
+ "title": "RAP for Statistics",
+ "section": "How to get started",
+ "text": "How to get started\nMeasure your publication against the RAP levels using our self assessment tool. This will give you a good starting point and initial points to work on to progress to the next level of RAP.\nOnce you’ve assessed your publication, have a look through our guidance below to narrow down how you can get started with improving those parts of your process.\nThe Statistics Development Team invites teams to take part in our partnership programme to develop their skills and implement RAP principles to a relevant project. Visit our page on getting started with the partnership programme for more details."
},
{
- "objectID": "learning-development/git.html#how-to-install-git",
- "href": "learning-development/git.html#how-to-install-git",
- "title": "Git",
- "section": "How to install Git",
- "text": "How to install Git\nDownload it from the Git website.\nGit doesn’t have an IDE, instead it will either integrate with your current IDE such as RStudio or Visual Studio Code, or run in the command line.\nWhen you first try to use Git you may be prompted for a GitHub username and password, if this happens you should generate a Personal Access Token (PAT) and use this as your password."
+ "objectID": "RAP/rap-statistics.html#all-source-data-stored-in-a-database",
+ "href": "RAP/rap-statistics.html#all-source-data-stored-in-a-database",
+ "title": "RAP for Statistics",
+ "section": "All source data stored in a database",
+ "text": "All source data stored in a database\n\n\n\nWhat does this mean?\nWhen we refer to ‘source data’, we take this to mean the data you use at the start of the process to create the underlying data files. Any cleaning at the end of a collection will happen before this.\nIn order for us to be able to have an end-to-end data pipeline where we can replicate our analysis across the department, we should store all of the raw data needed to create aggregate statistics in a managed Microsoft SQL Server. This includes any lookup tables and all administrative data from collections prior to any manual processing. This allows us to then match and join the data together in an end-to-end process using SQL queries.\nAs far as meeting the requirement to have all source data in a database, databases other than SQL may be acceptable, though we can’t support them in the same way.\nWhy do it?\nThe principle is that this source data will remain stable and is the point you can go back to and re-run the processes from if necessary. If for any reason the source data needs to change, your processes will be set up in a way that you can easily re-run them to get updated outputs based on the amended source data with minimal effort.\nSQL is a fantastic language for large scale data joining and manipulation; it allows us to replicate end-to-end from raw data to final aggregate statistics output. Having all the data in one place and processing it in one place makes our lives easier, and also helps us when auditing our work and ensuring reproducibility of results.\nHow to get started\n\nFor a collection of relevant resources to use when learning SQL, see our learning resources page, and for guidance on best practice when writing SQL queries, see the writing code and documentation sections on this page, as well as the guides immediately below on how to setup and use a SQL database.\n\n\nHow to set up a SQL working area\n\nThere are a few different options, depending on where you want your new area to exist. Visit our SQL learning page for details.\n\n\n\nMoving data to different areas\n\nIf your data is already in SQL, you can use this snippet of R code to move tables from one area (e.g. the iStore) to another (e.g. your team’s modelling area) to ensure all data are stored in a database.\nlibrary(odbc)\nlibrary(dplyr)\nlibrary(dbplyr)\nlibrary(DBI)\n\n# Step 1.1.: Connect to source server -------------------------------------------\ncon_source <- dbConnect(odbc(),\n Driver = \"SQL Server Native Client 11.0\",\n Server = \"Name_of_source_server\",\n Database = \"Source_database\",\n Trusted_Connection = \"yes\"\n)\n\n# Step 1.2.: Connect to target server\ncon_target <- dbConnect(odbc(),\n Driver = \"SQL Server Native Client 11.0\",\n Server = \"Name_of_target_server\",\n Database = \"Your_target_database\",\n Trusted_Connection = \"yes\"\n)\n\n# Step 2.1.: Pull the table from the source database\ntable_for_transfer <- tbl(con_source,in_schema(\"schema_name\", \"table_name\")) %>% collect()\n\n# Step 2.2.: Copy table into target database \ndbWriteTable(con_target,\"whatever_you_want_to_call_new_table\", table_for_transfer)\n\n\n\n\nImporting data to SQL Server\n\nThere’s lots of guidance online of how to import flat files from shared areas into Microsoft SQL server on the internet, including this guide.\nRemember that it is important to import them with consistent, thought-through naming conventions. You will thank yourself later.\n\n\n\nHow to grant access to your area\n\nMuch like setting up a SQL area, there are different ways to do this depending on the server your database is in. Visit our SQL learning page for details."
},
{
- "objectID": "learning-development/git.html#best-places-to-start",
- "href": "learning-development/git.html#best-places-to-start",
- "title": "Git",
- "section": "Best places to start",
- "text": "Best places to start\n\nContact us about attending one of our in-person Git workshops\nIf you’re new to Git and are unsure of what it does, then take a look through these Git for humans slides\nDavid Sands’ guide to getting started with Git is a helpful place to start.\nGooey Git by David Sands, provides a very neat overview of using git with R."
+ "objectID": "RAP/rap-statistics.html#standardised-reference-data",
+ "href": "RAP/rap-statistics.html#standardised-reference-data",
+ "title": "RAP for Statistics",
+ "section": "Standardised reference data",
+ "text": "Standardised reference data\nThere will of course be cases where some of the data you use is reference data not owned by the DfE, or is available online for you to download rather than in an internal server. There are ways of incorporating this into a reproducible analytical pipeline nicely, sometimes you can even use links/URLs in your code that will always pull the latest data such that you will never need to change the link to reflect updates to the data!\n\n\nRaw files on GitHub\n\nIf you want to use reference data from GitHub, you can use the URL to the raw file. A common example of when you might want to use this would be to use geographic lookup tables that contain names and codes of different geographic levels for EES files (which are available in our data screener repository).\nIf you find the data you’re interested in within a repository, rather than copying, cloning or downloading the data, you should click the ‘raw’ button (see the below screenshot).\n\n\n\nThis should take you to a new window which contains the raw data from the CSV file. Copy the URL from this page, as this is what we can use to pull the latest data into our code.\nYou can now use this URL as you would use a file path in a read.csv() query. For example;\ndevolved_area_lookup <- read.csv(\"https://raw.githubusercontent.com/dfe-analytical-services/dfe-published-data-qa/master/data/english_devolved_areas.csv\")\nThe above code snippet will load in the data as it would with any other CSV, however the benefit is if that data file is updated on GitHub, when you run the code it will always pull the latest version. This is especially useful for look up tables and reference data, and is best RAP practice as it removed the need for any manual steps (like downloading, copying & pasting or manually updating the data/code)!"
},
{
- "objectID": "learning-development/git.html#how-to-work-with-git",
- "href": "learning-development/git.html#how-to-work-with-git",
- "title": "Git",
- "section": "How to work with git",
- "text": "How to work with git\n\nGit Bash\n\nGit Bash allows you to run git commands without opening another IDE. You’d often need to use Git Bash to set your user settings, amend your proxy settings and clone repositories.\n\n\n\nGit with RStudio\n\nGit with R studio is a neat user interface for git. You don’t need to use any git bash commands, and everything is done using point and click. This is useful for day-to-day version control, but does not support the full functionality of git.\nHowever, you can still run the full suite of git commands by simply typing them in the “Terminal” of RStudio."
+ "objectID": "RAP/rap-statistics.html#processing-is-done-with-code",
+ "href": "RAP/rap-statistics.html#processing-is-done-with-code",
+ "title": "RAP for Statistics",
+ "section": "Processing is done with code",
+ "text": "Processing is done with code\n\n\n\nWhat does this mean?\nAll extraction, and processing of data should be done using code, avoiding any manual steps and moving away from a reliance on Excel, SPSS, and other manual processing. In order to carry out our jobs to the best of our ability it is imperative that we use the appropriate tools for the work that we do.\nEven steps such as copy and pasting data, or pointing and clicking, are fraught with danger, and these risks should be minimised by using code to document and execute these processes instead.\nWhy do it?\nUsing code brings numerous benefits, computers are far quicker, more accurate, and far more reliable than humans in many of the tasks that we do. Writing out these instructions saves us significant amounts of time, particularly when it can be reused in future years, or even next week when one specific number in the source file suddenly changes, and also provides us with editable documentation for our production processes, saving the need for writing down information in extra documents.\nReliability is a huge benefit of the automation that RAP brings - when one of the lines of data has to be amended a week before publication, it’s a life saver to know that you can re-run your process in minutes, and reassuring to know that it will give you the result you want. You can run the same code 100 times, and be confident that it will follow the same steps in the same order every single time.\nHow to get started\nSee our learning resources for a wealth of resources on SQL and R to learn the skills required to translate your process into code.\nThere are also two sections below with examples of tidying data in SQL and R to get you started.\nEnsure that any last-minute fixes to the process are written in the code and not done with manual changes.\n\n\nProducing tidy underlying data in SQL\n\nTo get started, here is a SQL query that you can run on your own machine and walks you through the basics of tidying a simple example dataset in SQL.\n\n\n\nTidying and processing data in R\n\nHere is a video of Hadley Wickham talking about how to tidy your data to these principles in R. This covers useful functions and how to complete common data tidying tasks in R. Also worth taking a look at applied data tidying in R, by RStudio.\nUsing the %>% pipe in R can be incredibly powerful, and make your code much easier to follow, as well as more efficient. If you aren’t yet familiar with this, have a look at this article that provides a useful beginners guide to piping and the kinds of functions you can use it for. The possibilities stretch about as far as your imagination, and if you have a function or task you want to do within a pipe, googling ‘how do I do X in dplyr r’ will usually start to point you in the right direction, alternatively you can contact us, and we’ll be happy to help you figure out how to do what you need.\nA quick example of how powerful this is is below, where my_data is processed to create new columns, have column names renamed, have the column names tidied using the janitor package, blank rows and columns removed, data filtered to only include specific geographic levels, and rows rearranged in order, all in a few lines of easy to follow code:\n\nprocessed_regional_data <- my_data %>% \n mutate(newPercentageColumn = (numberColumn / totalPopulationColumn) * 100) %>% \n rename(newPercentageColumn = percentageRate,\n numberColumn = number,\n totalPopulationColumn = population) %>% \n clean_names() %>% \n remove_empty() %>% \n filter(geographic_level == \"Regional\") %>% \n arrange(time_period, region_name)\n\nHelpful new functions in the tidyverse packages can help you to easily transform data from wide to long format (see tip 2 in the linked article for this, as it is often required for tidy data), as well as providing you with tools to allow you quickly and efficiently change the structure of your variables.\nFor further resources on learning R so that you’re able to apply it to your everyday work, have a look at the learning resources page."
},
{
- "objectID": "learning-development/git.html#quick-reference-lookup",
- "href": "learning-development/git.html#quick-reference-lookup",
- "title": "Git",
- "section": "Quick reference lookup",
- "text": "Quick reference lookup\n\nGitHub have created a cheat sheet for git commands."
+ "objectID": "RAP/rap-statistics.html#appropriate-tools",
+ "href": "RAP/rap-statistics.html#appropriate-tools",
+ "title": "RAP for Statistics",
+ "section": "Appropriate tools",
+ "text": "Appropriate tools\n\n\n\nWhat does this mean?\nUsing the recommended tools on our learning page (SQL, R and Git), or other suitable alternatives that allow you to meet the core principles. Ideally any tools used would be open source, Python is a good example of a tool that would also be well suited, though is less widely used in DfE and has a steeper learning curve than R.\nOpen-source refers to something people can modify and share because its design is publicly accessible. For more information, take a look at this explanation of open-source, as well as this guide to working in an open-source way. In practical terms, this means moving away from the likes of SPSS, SASS and Excel VBA, and utilising the likes of R or Python, version controlled with git, and hosted in a publicly accessible repository.\nWhy do it?\nThere are many reasons why we have recommended the tools that we have, the recommended tools are:\n\nalready in use at the department and easy for us to access\neasy and free to learn\ndesigned for the work that we do\nused widely across data science in both the public and private sector\nallow us to meet best practice when applying RAP to our processes\n\nHow to get started\nGo to our learning page to read more about the recommended tools for the jobs we do, as well as looking at the resources available there for how to build capability in them. Always feel free to contact us if you have any specific questions or would like help in understanding how to use those tools in your work.\nBy following our guidance in saving versions of code in an Azure DevOps, we will then be able to mirror those repositories in a publicly available GitHub area."
},
{
- "objectID": "learning-development/git.html#other-resources",
- "href": "learning-development/git.html#other-resources",
- "title": "Git",
- "section": "Other resources",
- "text": "Other resources\n\nAvison Ho and Linda Bennett gave this coffee and coding presentation on version controlling SQL with Git.\nHappy Git is a useful (though detailed) guide to setting up and using git.\nAdam Robinson and Zach Waller have produced guidance for how to use git in Azure DevOps (formally VSTS), which gives a detailed guide on how to use version control software in DfE analysis.\nWhile also mentioned above as a resource for learning R, chapter 6 of ESFA’s guide to R and Git is also worth looking at for Git alone.\nMicrosoft have produced documentation on using Git within AzureDevOps.\nFor those wanting to go deeper into understand the variety of git commands and what they do, there is a great online visual resource.\nWe also have a number of helpful sections on using git in practice at the end of our RAP page."
+ "objectID": "RAP/rap-statistics.html#using-run-scripts",
+ "href": "RAP/rap-statistics.html#using-run-scripts",
+ "title": "RAP for Statistics",
+ "section": "Using ‘run’ scripts",
+ "text": "Using ‘run’ scripts\nUtilising a single ‘run’ script to execute processes written in other scripts brings a number of benefits. It isn’t just about removing the need to manually trigger different code scripts to get the outputs, but it means the entire process, from start to finish, is fully documented in one place. This has a huge number of benefits, particularly for enabling new team members to pick up existing work quickly, without wasting time struggling to understand what has been done in the past.\n\n\nConnecting R to SQL\n\nIn order to create a single script to run all processes from, it is likely that you will need to use R to run SQL queries. If you are unsure of how to do this, take a look at the materials from Cathy Atkinson’s coffee and coding session on connecting R to SQL using DBI and odbc.\nChris Mason-Thom did another coffee and coding session on this, which you can watch below:\n\n\n\n\n\n\n\nDataset production scripts\n\n\n\n\nWhat does this mean?\nEach dataset can be created by running a single script, which may ‘source’ multiple scripts within it. This does not mean that all of the code to create a file must be written in a single script, but instead that there is a single ‘create file’ or ‘run’ script that sources every step in the correct order such that every step from beginning to end will be executed if you run that single ‘run’ script.\nThis ‘run’ script should take the source data right through to final output at the push of a button, including any manipulation, aggregation, suppression etc.\nWhy do it?\nHaving a script that documents the whole process for this saves time when needing to rerun processes, and provides a clear documentation of how a file is produced.\nHow to get started\nReview your current process - how many file scripts does it take to get from source data to final output, why are they separated, and what order should they be run in? Do you still have manual steps that could introduce human error (for example, manually moving column orders around in excel)?\nYou should automate any manual steps such as the example above. If it makes sense to, you could combine certain scripts to reduce the number. You can then write code in R to execute your scripts in order, so you are still only running one script to get the final output.\n\n\n\nWhole publication production scripts\n\n\n\n\nWhat does this mean?\nThe ultimate aim is to utilise a single script to document and run off everything for a publication, the data files, any QA, any summary reports. This script should allow you to run individual outputs by themselves as well, so make sure that each data file can be run in isolation by running single lines of this script. All quality assurance for a file is also included in the single script that can be used to create a file from source data (see the dataset production scripts section)\nWhy do it?\nThis carries all of the same benefits as having a single ‘run’ script for a file, but at a wider publication level, effectively documenting the entire publication process in one place. This makes it easier for new analysts to pick up the process, as well as making it quicker and easier to rerun as all reports relating to that file are immediately available if you ever make changes file.\nHow to get started\nThe Education, Health and Care Plans production cycle is a good example of a single publication ‘run’ script. They have kept their actual data processing in SQL, but all the running and manipulation of the data happens in R.\nThe cycle originally consisted of multiple SQL scripts, manual QA and generation of final files.\n\n\n\nThe team now have their end-to-end process fully documented, which can be run off of one single R script. The ‘run’ script points at the SQL scripts to run them all in one go, and also creates a QA report and corresponding metadata files that pass the data screener. Each data file can still be run in isolation from this script."
},
{
- "objectID": "learning-development/git.html#tips-for-using-git",
- "href": "learning-development/git.html#tips-for-using-git",
- "title": "Git",
- "section": "Tips for using Git",
- "text": "Tips for using Git\n\nBranches\n\nDavid Sands has produced a very helpful video on how to use branches in git, which also covers how to tackle merge conflicts if and when they arise:\n\n\n\n\nWhen you create a git project, it will automatically create a “main” (sometimes “master”) branch for you. This is where code that has been QA’d and you are happy with should sit.\n\nIt is good practice to have at least one other branch, we tend to call it “development”. This is the branch where you will be doing most of your work. To open a new branch, navigate to “branches” and click on the blue box highlighted below\n\n\n\nHaving two separate branches means that if anything goes wrong in the “development” branch, the “main” or “master” branch is still unaffected and runs without issue. This lets you test and QA the code more thoroughly before merging into your main branch.\nWhen working on your project, make sure that you are in the right branch. You can check this by navigating to the “Git” tab in RStudio as demonstrated below.\n\n\n\n\n\nCommits\n\nOnce you are happy with changes and want them to be in the latest version of your branch for all of your team to see, you can push “commits” up.\n\nWhen you make a change to a file, this will pop up in the “Git” window of your R console. Select the files you want to commit by ticking the “staged” box next to them.\n\n\n\nThis will bring up a new window. Add a comment describing your additions/changes, and click commit. You will see all the staged files disappear. Then click “Push” to push the committed files up to the online repository for all to use.\n\n\n\nWhen another member of your team makes a commit and you want to pull this into your local area to check and work off the latest version, click on the blue “pull” button.\n\n\nCommitting regularly is strongly recommended as each commit is a saved point in time that you can easily roll back to if needed. If you want to know more about how to do this, see the reverting a commit section below.\n\n\n\nPull requests\n\nWhen you have got to a place with the code and your committed changes where you are happy for it to be QA’d, you can open a pull request. This gives your team a chance to QA your changes before merging the branches together.\n\nNavigate to “Pull Requests” in the “Repos” tab of Azure DevOps and click the blue “New pull request” button.\nThis will take you to a new window. Here, you can add:\n\nTitle, tell your team what has changed\nDescription, tell your reviewer what they should check\nReviewers, add multiple if needed.\n\nAs a reviewer, to approve a pull request, follow the link in your email and click “approve” in the blue box. When all reviewers are happy for this to be the new master branch, click “complete”.\n\nCreating pull requests in GitHub follows a broadly similar process and should be intuitive from the above steps for Azure DevOps.\n\n\n\nMerging a branch into another one\n\nIf you want to merge a branch into another without doing a full pull request, you can do this using a terminal.\nThis may happen if you are working on a feature branch, and want to merge the latest changes to the master branch back into your feature branch before opening a PR to master with your own changes. Often this can let you deal with nasty conflicts up front, or allow you to keep working on your feature but update to have something new that’s on master.\nStart off by checking out your desired target branch - git checkout mybranch. Then merge in the branch you want (e.g. master) - git merge master.\n\n\n\nCherry picking\n\nIf you want to cherry pick specific commits for PR, you can do this by cherry picking the commits you want to use, and creating a new branch that has only those new commits that you want.\nTo start off you’ll need to identify the commits you want. In the terminal, run git log --oneline to get a log of commits for your current branch, use git log --online BRANCHNAME to specific the branch for the log. This gives a list of commit hashes and messages (stackoverflow response defining git hashes and commmit ID’s). You can press enter to get more commits, or q to quit\nThen go to the branch you want the commit to appear on and cherry pick your commits. Often, if this is to hop around something on a development to master pr, you would create a second development branch, cherry pick commits to there, and then PR that to master and delete the branch after merging.\nOn the branch you want to PR (i.e. your copy of a development branch purely for merging only cherry picked commits) run git cherry-pick COMMITHASH to add the specified commit, or git cherry-pick HASH1^..HASH4 for a specified list of commits (inclusive).\nHappy days!\n\n\n\nVisualising your tree\n\nYou can use gitk --all to visualise a tree of all previous commits up to this point.\n\n\n\n\nGetting commit IDs\n\nCommit IDs are the way Git identifies unique commits. They’re really helpful if you ever need to revert back to a previous commit if you’ve made a mistake.\nThere are lots of different ways to find out commit IDs:\n\nVisit the repo in Azure Devops and go to the commit of interest. At the top of the page there is a commit ID you can copy.\nNavigate to the repo in your file explorer, then open up Git Bash and type\n\ngit log --pretty=format:\"%h - %an, %ar : %\nOR\ngit log --pretty=oneline\nThere are a number of customisable versions of this, more information is available on the Git website.\n\n\n\nReverting a commit\n\nMade a mistake and need to revert? No problem! Reverting commits in git creates a new commit reversing your accidental commits, bringing you back to an earlier point in your branch.\nFor rolling back on a branch, you should revert any changes so that you’re not erasing history others might have pulled/cloned. E.g. If you want to revert the last 3 commits on master:\ngit revert --no-commit master~3..master\nThe no commit argument means that it just makes the reverts locally, and you can then commit them all as a single revert commit. If you don’t use --no-commit then it will start doing individual revert commits for you for each commit you’re reverting.\nYou can also revert back to a particular commit ID:\n\nNavigate to the repo in your file explorer, then open up Git Bash and type :\n\ngit revert [PASTE COMMIT ID HERE]\n\nThis opens up a window that asks you to write a commit message. You can skip this step as it automatically writes a revert message for you. Enter :wq which quits the writer window.\nYou can now push these changes to Azure Devops by entering git push into the git window\n\n\n\n\nTagging release versions\n\n\nIt can be useful to tag specific commits or releases at key points in time. For us a common example will be each publication cycle, to tag the version of the code used to process data for a particular release or amendment.\nGuidance on how to tag releases using git can be found on the draft version of the BPI code guidance\n\n\n\nFix: cannot resolve proxy\n\nIf you get this error when trying to pull / push to a repository from a DfE laptop:\nfatal: unable to access 'https://dfe-gov-uk.visualstudio.com/stats-development/_git/ees-analytics-new/': Could not resolve proxy: mwg.proxy.ad.hq.dept\nThen try running the following git command to clear the proxy settings in your git config file:\ngit config --global --unset http.proxy\n\n\n\nCleaning up local branches\n\nEssentially this finds all merged (old) branches, makes sure to not include master or development (can rename or add more if appropriate) and then runs those branches through the delete command. To be safe, run the first two thirds first, to print out the list of what you’ll be deleting:\ngit branch --merged | egrep -v \"(^\\*|master|development)\"\nThen if you’re happy, run the whole line:\ngit branch --merged | egrep -v \"(^\\*|master|development)\" | xargs git branch -d\nThat should be all local branches tidied up. Now to complete the job you can prune the tracking branches you have set up, usually this will just be:\ngit remote prune origin\nThough you can use git remote -v to find other remotes if you have them.\n\n\n\nCreating PATs\n\nPATs or Personal access tokens, are a preferred way to authenticate into repositories through code. Creating a PAT in GitHub is relatively intuitive.\nIn Azure DevOps, you can do this by following Microsoft’s documentation on creating a PAT in Azure DevOps.\n\n\n\nStoring secure variables\n\n\nAzure DevOps\n\nIn Azure DevOps you can securely store variables that are then used by your pipelines by making use of variable groups.\nFirst create a variable group by navigating to Pipelines > Library. Enter any variables you want to store here and make sure to change the variable type to secret if appropriate (i.e. login credentials or PATs).\n\nThen, in the pipeline you want to use the variables in, go to Variables > Variable groups and link the variable group as shown below. You can then call upon the variables as needed in your pipeline.\n\n\n\n\nGitHub\n\nIn GitHub you can store sensitive variables as encrypted secrets.\n\n\n\n\nMirroring a repository\n\nThere may be times when you have a repository in one place, such as Azure DevOps, but want to mirror the code and any changes to it in another place, such as GitHub. This can be for many different reasons, though commonly for us it will be to open up our code across government. Azure DevOps doesn’t provide us with easily publicly visible code repositories, however GitHub does.\nTo mirror a code repository from Azure DevOps, use the mirror git repository extension (already installed on dfe-gov-uk instance), make use of the PAT and secure variables sections above and then add a job to your pipeline and enter your parameters in the fields as per the example below:\n\n\n\n\nFixing the “JSON token 4” error\n\nThis error can appear when you are trying to push changes to Azure DevOps.\nIf it appears and you have not changed your password recently, try locking your laptop and seeing if on re-login it prompts you for a password change.\nIf you have changed changed your password recently, try signing out of Azure and then back in again."
+ "objectID": "RAP/rap-statistics.html#recyclable-code-for-future-use",
+ "href": "RAP/rap-statistics.html#recyclable-code-for-future-use",
+ "title": "RAP for Statistics",
+ "section": "Recyclable code for future use",
+ "text": "Recyclable code for future use\n\n\n\nWhat does this mean?\nWe’d expect that any recyclable code would take less than 30 minutes of editing before being able to run again in a future iteration of the publication.\nWhy do it?\nOne huge benefit that comes with using code in our processes, is that we can pick them up in future years and reuse with minimum effort, saving us huge amounts of resource. To be able to do this, we need to be conscious of how we write our code, and write it in a way that makes it easy to use in future releases for the publication.\nHow to get started\nReview your code and consider the following:\n\nWhat steps might need re-editing or could become irrelevant?\nCan you move all variables that require manual input (e.g. table names, years) to be assigned at the top of the code, so it’s easy to edit in one place with each iteration?\nAre there any fixed variables that are prone to changing such as geographic boundaries, that you could start preparing for changes now by making it easy to adapt in future?\n\nFor example, if you refer to the year of publication in your code a lot, consider replacing every instance with a named variable, which you only need to change once at the start of your code. In the example below, the year is set at the top of the code, and is used to define “prev_year”, both of which are used further down the code to filter the data based on year.\n\nthis_year <- 2020\nprev_year <- this_year - 1\n\ndata_filtered <- data %>% \n filter(year == this_year)\n\ndata_filtered_last_year <- data %>% \n filter(year == prev_year)"
},
{
- "objectID": "CONTRIBUTING.html",
- "href": "CONTRIBUTING.html",
- "title": "Contributing to the Analyst’s Guide",
- "section": "",
- "text": "Thank you for investing your time in contributing to the analyst’s guide! All contributions are very welcome - we want this to be a useful resource for the whole analytical community in DfE :sparkles:.\nRead our Code of Conduct to keep our community approachable and respectable.\nThe main branch of the site is protected and can only be updated via pull requests. All pull requests must be approved by a repository admin before merging.\nIn this guide you will get an overview of the contribution workflow from opening an issue, creating a PR, reviewing, and merging the PR.\n\n\nTo get an overview of the project, read the README. Here are some resources to help you get started with open source contributions:\n\nFinding ways to contribute to open source on GitHub\nSet up Git\nGitHub flow\nCollaborating with pull requests\n\n\n\nFor more information specific to how a quarto site works, and for examples of what is possible within a static quarto site, see the main Quarto website.\n\n\n\n\n\n\nIf you spot a problem with the site, search if an issue already exists. If a related issue doesn’t exist, you can open a new issue using a relevant issue form.\n\n\n\nScan through our existing issues to find one that interests you. If you find an issue to work on, you are welcome to open a PR with a fix.\n\n\n\n\nKey things to remember when making or proposing changes: - Where guidance already exists elsewhere it should be linked to rather than duplicated - When adding images, save them in the /images folder - When adding files for download, save them in the /resources folder - Don’t commit rendered .html files for pages - If your changes remove or edit a pre-existing anchor link, consider how that will be redirected for users who may have bookmarked it\n\n\nClick Edit this page at the bottom of the right hand table of contents of any page to make small changes such as a typo, sentence fix, or a broken link. This takes you to the .Qmd file where you can make your changes and create a pull request for a review.\n\n\n\n\nClone or fork the repository.\nOpen the repository in your editor of choice, e.g. R Studio.\nCreate a working branch and start with your changes!\nCommit and push the changes to your working branch.\n\n\n\n\n\nAll pull requests should be made against the main branch.\nWhen you’re finished with the changes, create a pull request, also known as a PR. - Don’t forget to link PR to issue if you are solving one.\nOnce you submit your PR, a repository admin will review your proposal. We may ask questions or request additional information. - We may ask for changes to be made before a PR can be merged, either using suggested changes or pull request comments. You can apply suggested changes directly through the UI. You can make any other changes and then commit them to your branch. - As you update your PR and apply changes, mark each conversation as resolved. - If you run into any merge issues, checkout this Git tutorial to help you resolve merge conflicts and other issues.\n\n\n\n\nIf you need any assistance at all, please contact statistics.development@education.gov.uk."
+ "objectID": "RAP/rap-statistics.html#standards-for-coding",
+ "href": "RAP/rap-statistics.html#standards-for-coding",
+ "title": "RAP for Statistics",
+ "section": "Standards for coding",
+ "text": "Standards for coding\nCode can be written in many different ways, and in languages such as R, there are often many different functions and routes that you can take to get to the same end result. On top of that, there are even more possibilities for how you can format the code. This section will take you through some widely used standards for coding to help bring standardisation to this area and make it easier to both write and use our code.\n\n\nClean final code\n\n\n\n\nWhat does this mean?\n\nThis code should meet the best practice standards below (for SQL and R). If you are using a different language, such as Python, then contact us for advice on the best standards to use when writing code.\nThere should be no redundant or duplicated code, even if this has been commented out. It should be removed from the files to prevent confusion further down the line.\nThe only comments left in the code should be those describing the decisions you have made to help other analysts (and future you) to understand your code. More guidance on commenting in code can be found later on this page.\n\nWhy do it?\nClean code is efficient, easy to write, easy to review, and easy to amend for future use. Below are some recommended standards to follow when writing code in SQL and R.\nHow to get started\nWatch the coffee and coding session introducing good code practice below:\n\n\n\n\nThen you should also watch the follow up intermediate session:\n\n\n\n\nClean code should include comments. Comment why you’ve made decisions, don’t comment what you are doing unless it is particularly complex as the code itself describes what you are doing. If in doubt, more comments are better than too few though. Ideally any specific comments or documentation should be alongside the code itself, rather than in separate documents.\n\n\nSQL\n\nFor best practice on writing SQL code, here is a particularly useful word document produced by our Data Hub. This outlines a variety of best practices, ranging from naming conventions, to to formatting your SQL code so that it is easy to follow visually.\n\n\n\nR\n\nWhen using R, it is generally best practice to use R projects as directories for your work.\nThe recommended standard for styling your code in R, is the tidyverse styling, which is fast becoming the global standard. What is even better is that you can automate this using the styler package, which will literally style your code for you at the click of a button, and is well worth a look.\n\n\n\nThere is also plenty of guidance around the internet for best practice when writing efficient R code.\n\n\n\nHTML\n\nIf you ever find yourself writing html, or creating it through rmarkdown, you can check your html using w3’s validator."
},
{
- "objectID": "CONTRIBUTING.html#new-contributor-guide",
- "href": "CONTRIBUTING.html#new-contributor-guide",
- "title": "Contributing to the Analyst’s Guide",
- "section": "",
- "text": "To get an overview of the project, read the README. Here are some resources to help you get started with open source contributions:\n\nFinding ways to contribute to open source on GitHub\nSet up Git\nGitHub flow\nCollaborating with pull requests\n\n\n\nFor more information specific to how a quarto site works, and for examples of what is possible within a static quarto site, see the main Quarto website.\n\n\n\n\n\n\nIf you spot a problem with the site, search if an issue already exists. If a related issue doesn’t exist, you can open a new issue using a relevant issue form.\n\n\n\nScan through our existing issues to find one that interests you. If you find an issue to work on, you are welcome to open a PR with a fix.\n\n\n\n\nKey things to remember when making or proposing changes: - Where guidance already exists elsewhere it should be linked to rather than duplicated - When adding images, save them in the /images folder - When adding files for download, save them in the /resources folder - Don’t commit rendered .html files for pages - If your changes remove or edit a pre-existing anchor link, consider how that will be redirected for users who may have bookmarked it\n\n\nClick Edit this page at the bottom of the right hand table of contents of any page to make small changes such as a typo, sentence fix, or a broken link. This takes you to the .Qmd file where you can make your changes and create a pull request for a review.\n\n\n\n\nClone or fork the repository.\nOpen the repository in your editor of choice, e.g. R Studio.\nCreate a working branch and start with your changes!\nCommit and push the changes to your working branch.\n\n\n\n\n\nAll pull requests should be made against the main branch.\nWhen you’re finished with the changes, create a pull request, also known as a PR. - Don’t forget to link PR to issue if you are solving one.\nOnce you submit your PR, a repository admin will review your proposal. We may ask questions or request additional information. - We may ask for changes to be made before a PR can be merged, either using suggested changes or pull request comments. You can apply suggested changes directly through the UI. You can make any other changes and then commit them to your branch. - As you update your PR and apply changes, mark each conversation as resolved. - If you run into any merge issues, checkout this Git tutorial to help you resolve merge conflicts and other issues."
+ "objectID": "RAP/rap-statistics.html#peer-reviewing-code",
+ "href": "RAP/rap-statistics.html#peer-reviewing-code",
+ "title": "RAP for Statistics",
+ "section": "Peer reviewing code",
+ "text": "Peer reviewing code\nPeer review is an important element of quality assuring our work. We often do it without realising by bouncing ideas off of one another and by getting others to ‘idiot check’ our work. When writing code, ensuring that we get our work formally peer reviewed is particularly important for ensuring it’s quality and value.\nPrior to receiving code for peer review, the author should ensure that all code files are clean, commented appropriately and for larger projects should be held in a repo with an appropriate README file.\nWhen peer reviewing code you should be consider the following questions -\n\nDoes the code do what the author intended?\nIf you’re able to run the code, does it run without errors? If warnings are displayed, are they explained?\nIf the project has unit/integration tests, do they pass?\nAre there any tests / checks that could be added into the code that would help to give greater confidence that it is doing what it is intended to?\nAre there comments explaining why any decisions have been made?\nIs the code written and structured sensibly?\nAre there any ways to make the code more efficient (either in number of lines or raw speed)?\nDoes the code follow best practice for styling and structure?\nAre there any other teams/bits of code you’re aware of that do similar things and would be useful to point the authors towards?\nAt the end of the review, was there any information you needed to ask about that should be made more apparent in the code or documentation?\n\nDepending on your access you may or may not be able to run the code yourself, but there should be enough information within the code and documentation to be able to respond to these questions.\n\n\nReview of code within team\n\n\n\n\nWhat does this mean?\n\nIs someone else in the team able to generate the same outputs?\nHas someone else in the team reviewed the code and given feedback?\nHave you taken on their feedback and improved the code?\n\nWhy do it?\nThere are many benefits to this, for example:\n\nEnsuring consistency across the team\nMinimizing mistakes and their impact\nEnsuring the requirements are met\nImproving code performance\nSharing of techniques and knowledge\n\nHow to get started\nIf you can’t answer yes, then:\n\nGet a member of the team to run the code using only your documentation\nUse their feedback to improve documentation/in-line comments in code\nOther tips for getting started with peer review can be found in the Duck Book\nThe Duck Book also contains some helpful code QA checklists to help get you thinking about what to check\n\n\n\nImproving code performance\n\nPeer reviewing code and not sure where to start? Improving code performance can be a great quick-win for many production teams. There will be cases where code you are reviewing does things in a slightly different way to how you would: profiling the R code with the microbenchmark package is a way to objectively figure out which method is more efficient.\nFor example below, we are testing out case_when, if_else and ifelse.\nmicrobenchmark::microbenchmark( \n case_when(1:1000 < 3 ~ \"low\", TRUE ~ \"high\"), \n if_else(1:1000 < 3, \"low\", \"high\"),\n ifelse(1:1000 < 3, \"low\", \"high\") \n)\nRunning the code outputs a table in the R console, giving profile stats for each expression. Here, it is clear that on average, if_else() is the fastest function for the job.\nUnit: microseconds\n expr min lq mean median uq max neval\n case_when(1:1000 < 3 ~ \"low\", TRUE ~ \"high\") 167.901 206.2510 372.7321 300.2515 420.1005 4187.001 100\n if_else(1:1000 < 3, \"low\", \"high\") 55.301 74.0010 125.8741 103.7015 138.3010 538.201 100\n ifelse(1:1000 < 3, \"low\", \"high\") 266.200 339.4505 466.7650 399.7010 637.6010 851.502 100\n\n\n\n\n\nReview of code from outside the team\n\n\n\n\nWhat does this mean?\n\nHas someone from outside of the team and publication area reviewed the code and given feedback?\nHave you taken on their feedback and improved the code?\n\nWhy do it?\nAll of the benefits you get from peer reviewing within your own team, multiple times over. Having someone external offers new perspectives, holds you to account by breaking down assumptions, and offers far greater opportunity for building capability through knowledge sharing.\nHow to get started\nWhile peer reviewing code within the team is often practical, having external analysts peer review your code can bring a fresh perspective. If you’re interested in this, please contact us, and we can help you to arrange someone external to your team to review your processes. For this to work smoothly, we recommend that your code is easily accessible for other analysts, such as hosted in an Azure DevOps repo and mirrored to github."
},
{
- "objectID": "CONTRIBUTING.html#support",
- "href": "CONTRIBUTING.html#support",
- "title": "Contributing to the Analyst’s Guide",
- "section": "",
- "text": "If you need any assistance at all, please contact statistics.development@education.gov.uk."
+ "objectID": "RAP/rap-statistics.html#automated-quality-assurance",
+ "href": "RAP/rap-statistics.html#automated-quality-assurance",
+ "title": "RAP for Statistics",
+ "section": "Automated quality assurance",
+ "text": "Automated quality assurance\nAny data files that have been created will need to be quality assured. These checks should be automated where possible, so the computer is doing the hard work - saving us time, and to ensure their reliability.\nSome teams are already making great progress with automated QA and realising the benefits of it. The Statistics Development Team are working with these to provide generalised code that teams can use as a starting point for automated QA. The intention is that teams can then run this as a minimum, before then looking to develop more area specific checks to the script and/or continue with current checking processes in tandem. If your team already use, or are working towards using, automated QA then get in touch as we’d be keen to see what you have.\nIt is assumed that when using R, automated scripts will output .html reports that the team can read through to understand their data and identify any issues, and save as a part of their process documentation.\nFor more information on general quality assurance best practice in DfE, see the How to QA guide.\n\n\nBasic automated QA\n\n\n\n\nWhat does this mean?\nThe list of basic automated QA checks, with code examples can be found below and in our GitHub repository:\n\nChecking for minimum, maximum, and average values across your data\nChecking for extreme values and outliers\nEnsuring there are no duplicate rows or duplicate columns\nChecking that where appropriate, geographical subtotals add up to totals (e.g. all the numeric values for LAs in Yorkshire and The Humber add up to the regional total)\nBasic trend analysis using scatter plots, to help you spot outliers and help tell the story of your data.\n\nThe Statistics Development Team have developed the QA app to include some of these basic QA outputs.\nWhy do it?\nQuality is one of the three pillars that our code of practice is built upon. These basic level checks allow us to have confidence that we are accurately processing the data.\nAutomating these checks ensures their accuracy and reliability, as well as being dramatically quicker than doing these manually.\nHow to get started\nTry using our template code snippets to get an idea of how you could automate QA of your own publication files. A recording of our introduction to automated QA is also available at the top of the page.\n\n\n\nPublication specific automated QA\n\n\n\n\nWhat does this mean?\nMany teams will have aspects of their data and processes that require Quality Assuring beyond the generalisable basic checks above. Therefore it is expected that teams develop their own automated QA checks to QA specificities of their publications not covered by the basic checks.\nWhy do it?\nQuality is one of the three pillars that our code of practice is built upon. By building upon the basic checks to develop bespoke QA for our publications, we can increase our confidence in the quality of the processes and outputs that they produce.\nHow to get started\nWe expect that the basic level of automated QA will cover most needs that publication teams have. However, we also expect that each publication will have it’s own quirks that require a more bespoke approach. An example of a publication with it’s own bespoke QA checks will appear in this space shortly. For the time being, try to consider what things you’d usually check as flags that something hasn’t gone right with your data. What are the unique aspects of your publication’s data, and how can you automate checks against them to give you confidence in it’s accuracy and reliability?\nFor those who are interested in starting writing their own QA scripts, it’s worth looking at packages in R such as testthat, including the coffee and coding talk on it by Peter Curtis, as well as this guide on testing by Hadley Wickham.\nThe janitor package in R also has some particularly useful functions, such as clean_names() to automatically clean up your variable names, remove_empty() to remove any completely empty rows and columns, and get_dupes() which retrieves any duplicate rows in your data - this last one is particularly powerful as you can feed it specific columns and see if there’s any duplicate instances of values across those columns."
},
{
- "objectID": "CODE_OF_CONDUCT.html",
- "href": "CODE_OF_CONDUCT.html",
- "title": "Contributor Covenant Code of Conduct",
- "section": "",
- "text": "We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.\nWe pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.\n\n\n\nExamples of behavior that contributes to a positive environment for our community include:\n\nDemonstrating empathy and kindness toward other people\nBeing respectful of differing opinions, viewpoints, and experiences\nGiving and gracefully accepting constructive feedback\nAccepting responsibility and apologizing to those affected by our mistakes, and learning from the experience\nFocusing on what is best not just for us as individuals, but for the overall community\n\nExamples of unacceptable behavior include:\n\nThe use of sexualized language or imagery, and sexual attention or advances of any kind\nTrolling, insulting or derogatory comments, and personal or political attacks\nPublic or private harassment\nPublishing others’ private information, such as a physical or email address, without their explicit permission\nOther conduct which could reasonably be considered inappropriate in a professional setting\n\n\n\n\nCommunity leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.\nCommunity leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.\n\n\n\nThis Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.\n\n\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at statistics.development@education.gov.uk. All complaints will be reviewed and investigated promptly and fairly.\nAll community leaders are obligated to respect the privacy and security of the reporter of any incident.\n\n\n\nCommunity leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:\n\n\nCommunity Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.\nConsequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.\n\n\n\nCommunity Impact: A violation through a single incident or series of actions.\nConsequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.\n\n\n\nCommunity Impact: A serious violation of community standards, including sustained inappropriate behavior.\nConsequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.\n\n\n\nCommunity Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.\nConsequence: A permanent ban from any sort of public interaction within the community.\n\n\n\n\nThis Code of Conduct is adapted from the Contributor Covenant, version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.\nCommunity Impact Guidelines were inspired by Mozilla’s code of conduct enforcement ladder.\nFor answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations."
+ "objectID": "RAP/rap-statistics.html#automating-summary-statistics",
+ "href": "RAP/rap-statistics.html#automating-summary-statistics",
+ "title": "RAP for Statistics",
+ "section": "Automating summary statistics",
+ "text": "Automating summary statistics\nAs a part of automating QA, we should also be looking to automate the production of summary statistics alongside the tidy underlying data files, this then provides us with instant insight into the stories underneath the numbers.\n\n\nAutomated summaries\n\n\n\n\nWhat does this mean?\nSummary outputs are automated and used to explore the stories of the data.\nThe Statistics Development Team have developed the QA app to include some of these automated summaries, including minimum, maximum and average summaries for each indicator.\nAt a basic level we want teams to make use of the QA app to explore their data:\n\nHave you used the outputs of the automated QA from the screener to understand the data?\nRun automated QA, ensure that all interesting outputs/trends are reflected in the accompanying text\n\nWhy do it?\nValue is one of the three pillars of our code of practice. Even more specifically it states that ‘Statistics and data should be presented clearly, explained meaningfully and provide authoritative insights that serve the public good.’.\nAs a result, we should be developing automated summaries to help us to better understand the story of the data and be authoritative and rigorous in our telling of it.\nHow to get started\nConsider:\n\nUse the additional tabs available after a data file passes the data screener as a starting point to explore trends across breakdowns and years.\nRunning your publication-specific automated QA, ensuring that all interesting outputs/trends are reflected in the accompanying text\n\n\n\n\nPublication specific automated summaries\n\n\n\n\nWhat does this mean?\n\nHave you gone beyond the outputs of the QA app to consider automating further insights for your publication specifically? E.g. year on year changes for specific measures, comparisons of different characteristics that are of interest to the general public\nAre you using these outputs to write your commentary?\n\nWhy do it?\nAll publications are different, and therefore it is important that for each publication, teams go beyond the basics and produce automated summaries specific to their area.\nHow to get started\nConsider:\n\nIntegrating extra publication-specific QA into the production process\nConsider outputs specific to your publication that would help you to write commentary/draw out interesting analysis"
},
{
- "objectID": "CODE_OF_CONDUCT.html#our-pledge",
- "href": "CODE_OF_CONDUCT.html#our-pledge",
- "title": "Contributor Covenant Code of Conduct",
- "section": "",
- "text": "We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.\nWe pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community."
+ "objectID": "RAP/rap-statistics.html#sensible-folder-and-file-structure",
+ "href": "RAP/rap-statistics.html#sensible-folder-and-file-structure",
+ "title": "RAP for Statistics",
+ "section": "Sensible folder and file structure",
+ "text": "Sensible folder and file structure\n\n\n\nWhat does this mean?\nAs a minimum you should have a folder that includes all of the final versions of documents produced and published, per release, within a folder for the wider publication. Ask yourself if it would be easy for someone who isn’t in the team to find specific files, and if not, is there a better way that you could name and structure your folders to make them more intuitive to navigate?\nWhy do it?\nHow you organize and name your files will have a big impact on your ability to find those files later and to understand what they contain. You should be consistent and descriptive in naming and organizing files so that it is obvious where to find specific data and what the files contain.\nHow to get started\nSome questions to help you consider whether your folder structure is sensible are:\n\nAre all documentation, code and outputs for the publication saved in one folder area?\nIs simple version control clearly applied (e.g. having all final files in a folder named “final”?\nAre there sub-folders like ‘code’, ‘documentation’‘, ’outputs’ and ‘final’ to save the relevant working files in?\nAre you keeping a version log up to date with any changes made to files in this final folder?\n\n\n\nNaming conventions\n\nHaving a clear and consistent naming convention for your files is critical. Remember that file names should:\nBe machine readable\n\nAvoid spaces.\nAvoid special characters such as: ~ ! @ # $ % ^ & * ( ) ` ; < > ? , [ ] { } ‘ “.\nBe as short as practicable; overly long names do not work well with all types of software.\n\nBe human readable\n\nBe easy to understand the contents from the name.\n\nPlay well with default ordering\n\nOften (though not always!) you should have numbers first, particularly if your file names include dates.\nFollow the ISO 8601 date standard (YYYYMMDD) to ensure that all of your files stay in chronological order.\nUse leading zeros to left pad numbers and ensure files sort properly, avoiding 1,10,2,3.\n\nIf in doubt, take a look at this presentation, or this naming convention guide by Stanford, for examples reinforcing the above."
},
{
- "objectID": "CODE_OF_CONDUCT.html#our-standards",
- "href": "CODE_OF_CONDUCT.html#our-standards",
- "title": "Contributor Covenant Code of Conduct",
- "section": "",
- "text": "Examples of behavior that contributes to a positive environment for our community include:\n\nDemonstrating empathy and kindness toward other people\nBeing respectful of differing opinions, viewpoints, and experiences\nGiving and gracefully accepting constructive feedback\nAccepting responsibility and apologizing to those affected by our mistakes, and learning from the experience\nFocusing on what is best not just for us as individuals, but for the overall community\n\nExamples of unacceptable behavior include:\n\nThe use of sexualized language or imagery, and sexual attention or advances of any kind\nTrolling, insulting or derogatory comments, and personal or political attacks\nPublic or private harassment\nPublishing others’ private information, such as a physical or email address, without their explicit permission\nOther conduct which could reasonably be considered inappropriate in a professional setting"
+ "objectID": "RAP/rap-statistics.html#documentation",
+ "href": "RAP/rap-statistics.html#documentation",
+ "title": "RAP for Statistics",
+ "section": "Documentation",
+ "text": "Documentation\n\n\n\nWhat does this mean?\n\nYou should be annotating as you go, ensuring that every process and decision made is written down. Processes are ideally written with code, and decisions in comments.\nThere should be a README notes file, that clearly details the steps in the process, any dependencies (such as places where access needs to be requested to) and how to carry out the process.\nAny specialist terms should also be defined if required (e.g. The NFTYPE lookup can be found in xxxxx. “NFTYPE” means school type).\n\nWhy do it?\nWhen documenting your processes you should leave nothing to chance, we all have wasted time in the past trying to work out what it was that we had done before, and that time increases even more when we are picking up someone else’s work. Thorough documentation saves us time, and provides a clear audit trail of what we do. This is key for the ‘Reproducible’ part of RAP, our processes must be easily reproducible and clear documentation is fundamental to that.\nHow to get started\nTake a look at your processes and be critical - could another analyst pick them up without you there to help them? If the answer is no (don’t feel ashamed, it will be for many teams) then go through and note down areas that require improvement, so that you can revise them with your team.\nTake a look at the sections below for further guidance on improving your documentation.\n\n\nCommenting in code\n\nWhen writing code, whether that is SQL, R, or something else, make sure you’re commenting as you go. Start off every file by outlining the date, author, purpose, and if applicable, the structure of the file, like this:\n----------------------------------------------------------------------------------------------\n-- Script Name: Section 251 Table A 2019 - s251_tA_2019.sql\n-- Description: Extraction of data from IStore and production of underlying data file\n-- Author: Cam Race\n-- Creation Date: 15/11/2019\n----------------------------------------------------------------------------------------------\n\n----------------------------------------------------------------------------------------------\n--// Process\n-- 1. Extract the data for each available year\n-- 2. Match in extra geographical information\n-- 3. Create aggregations - both categorical and geographical totals\n-- 4. Tidy up and output results\n-- 5. Metadata creation\n----------------------------------------------------------------------------------------------\nCommented lines should begin with – (SQL) or # (R), followed by one space and your comment. Remember that comments should explain the why, not the what.\nIn SQL you can also use /** and **/ to bookend comments over multiple lines.\nIn rmarkdown documents you can bookend comments by using <!-- and -->.\nUse commented lines of - to break up your files into scannable chunks based upon the structure and subheadings, like the R example below:\n# Importing the data -----------------------------------------------------------------------------------\n\nDoing this can visually break up your code into sections that are easy to navigate around. It will also add that section to your outline, which can be used in RStudio using Ctrl-Shift-O. More details on the possibilities for this can be found in the RStudio guidance on folding and sectioning code.\nYou might be thinking that it would be nice if there was software that could help you with documentation, if so, read on, as Git is an incredibly powerful tool that can help us easily and thoroughly document versions of our files. If you’re at the stage where you are developing your own functions and packages in R, then take a look at roxygen2 as well.\n\n\n\nWriting a README file\n\nWhat does this mean?\nA README is a text file (.txt) that introduces and explains a project. It contains information that is required to understand what the project is about and how to use it.\nWhy do it?\nIt’s an easy way to answer questions that your audience will likely have regarding how to install and use your project and also how to collaborate with you.\nHow to get started\nAs a starting point, you should aim to have as many of the following sections as are applicable to your project:\n\nIntroduction\nRequirements (access, software, skills/knowledge)\nHow to use\nHow to contribute\nContact details\n\nThe Self-assessment tool and the QA app give two examples of readme files structured like this."
},
{
- "objectID": "CODE_OF_CONDUCT.html#enforcement-responsibilities",
- "href": "CODE_OF_CONDUCT.html#enforcement-responsibilities",
- "title": "Contributor Covenant Code of Conduct",
- "section": "",
- "text": "Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.\nCommunity leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate."
+ "objectID": "RAP/rap-statistics.html#version-control-with-git",
+ "href": "RAP/rap-statistics.html#version-control-with-git",
+ "title": "RAP for Statistics",
+ "section": "Version control with git",
+ "text": "Version control with git\nIf you do not already have git downloaded, you can download the latest version from their website.\nFor now, take a look at at the resources for learning Git on the learning resources page.\n\n\nVersion controlled final code scripts\n\n\n\n\nWhat does this mean?\nThis means having the final copies of code and documentation saved in a git-controlled Azure DevOps repo in the official-statistics-production area.\nWhy do it?\nHaving the final copy of the scripts version controlled gives assurance around how the data was created. It also allows teams to easily record any last minute changes to the code after the initial final version by using the version control to log this.\nHow to get started\nThe first step is to get your final versions of code and documentation together in a single folder.\nWe have a specific area set up for you to host your publication code in on the dfe-gov-uk instance of Azure DevOps, entitled official-statistics-production.\nTo gain access to this area, please raise a request on service desk by navigating through the pages detailed in the animation below.\n\n\n\nOnce you have navigated to this page, fill out the form with the following details and send your request off.\n\n\n\nAccess is usually granted within a few working days. Alert the Statistics Development Team when this is confirmed, and we will set up your repository and give your team access.\n\n\n\nUse open source repositories\n\n\n\n\nWhat does this mean?\nSaving or cloning your work into a repository that is visible to the public. We currently have brilliant examples of this in our DfE analytical services GitHub area, in which all of the code used to create public dashboards is publicly available.\nFor statistics publications we expect teams to be able to mirror their proccess code on GitHub after publication, which will help open up their code for other analysts to learn from.\nWe are currently working on ways to mirror private repos (i.e. AzureDevOps) to public repos on publication of your data. If you are interested in this please contact the Statistics Development Team.\nWhy do it?\nIt’s a key part of the technology code of practice as an agreed standard for digital services across government.\nHow to get started?\nContact us to get a repository set up in Azure DevOps, to set up a mirroring process to GitHub or to set up a repository on our dfe-analytical-services area on GitHub.\nYou should consider the following principles making an Official Statistics production repository public (some examples are R specific though can be applied to other languages):\n\nFollow the guidance on writing a readme file, and add context in about what Official/National statistics are\nEnsure no data (either input or output) is included in the repository\nHave a clear and organised folder structure (such as having R scripts in an ‘R’ folder)\nCheck your code is styled according the tidyverse styling\nUse renv for package management\nUse an R project\n\nWhen naming your publication’s repository you should use the publication name fully written out, in lower case, and with dashes for spaces – ‘graduate-labour-market-statistics’.\nA single repository should be used for all releases of your publication, there’s no need to have multiple as all the history is saved within previous commits. You can make use of tagging releases in git to help differentiate between each cycle.\n\n\nAvoid revealing sensitive information\n\nHere are some general best practice tips:\n\nUsing .gitignore to ignore files and folders to prevent committing anything sensitive\nNever committing outputs unless they’ve been checked over, even aggregates. We suggest only outputting to an output folder which is in the .gitignore file, to ensure this doesn’t happen by mistake\nKeeping datasets and secrets (e.g. API keys) outside the repository as much as possible, make use of secure variables\nChecking git histories: if someone is planning on open-sourcing code that has previously been in a private repository or only version-controlled locally, you want to be careful not to have anything sensitive in the commit history. You can do this by following the above rules. When in doubt, you can remove the git history and start the public repo without it\nYou can remove a file from the entire commit history if you did commit anything sensitive, although you still need to follow the usual procedures if this was a data breach\n\nYou can find out more in the duck book’s guidance on using git.\n\n\n\n\nCollaboratively develop code using git\n\n\n\n\nWhat does this mean?\n\nHas code development taken place in git, collaboratively across the team?\nAre you making use of pull requests for team members to review and comment on code updates?\nIs there a clear paper trail of changes to code (commits)?\n\nWhy do it?\nUsing git allows multiple people to simultaneously develop the same code using branches, all with a crystal clear audit trail showing what changes were made when using commits. It makes it easy for team members to review changes via pull requests.\nHow to get started\nTo get started you should:\n\n\nGet your code into a git controlled folder\n\nGet code into a git controlled folder in whatever version it is currently in. Use the following steps to do so:\n\nOpen the folder where your project is saved, right click anywhere in that window, and click “Git Bash Here”.\nThis will open a black box. Type in the following and hit enter\n\n\ngit init\n\n\nAfter hitting enter, type in the following and hit enter again after each line. You will need the URL of your Azure DevOps repository to complete this step. Contact the Statistics Development Team if you are not sure what this is or do not have one.\n\n\ngit add .\n\ngit commit -m \"first commit\"\n\ngit remote add origin YOUR_URL_HERE\n\ngit push -f origin --all\n\n\nYou may be prompted for either your windows or git credentials at this stage.\n\nIf prompted for your windows credentials, enter the username and password combination you use to log into your DfE device.\nIf prompted for your git credentials, visit your online repository, click on the blue “clone” box, and click “generate git credentials”. This will generate a username and password for you to enter.\n\n\n\nVisit your repository online, and check that all the files have uploaded. Other members of your team will now be able to work from your code.\n\n\n\n\nBuild capability within the team\n\n\nEnsure all team members have access to your project in the Azure DevOps official-statistics-production area. Contact the Statistics Development Team if there are any issues.\nGet team members to clone your repository in to their personal area, so everyone is able to work on code at the same time.\n\nTo clone code, they will need to do the following:\n\nRun through steps 1 - 2a of getting a file into a git controlled folder\nAfter running those lines, type in the following with your repository URL in the “YOUR_URL_HERE” space. This will clone the online repository to your local area.\n\n\ngit clone YOUR_URL_HERE\n\n\nMake use of git and version control in your team projects regularly. Like learning anything new, putting it into practice regularly is the best way to become confident in using it.\n\nPlease refer to the other links on the Git learning resources page to learn more about how to use git in practice."
},
{
- "objectID": "CODE_OF_CONDUCT.html#scope",
- "href": "CODE_OF_CONDUCT.html#scope",
- "title": "Contributor Covenant Code of Conduct",
+ "objectID": "writing-visualising/cd.html",
+ "href": "writing-visualising/cd.html",
+ "title": "Writing and visualising",
"section": "",
- "text": "This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event."
+ "text": "Things to consider when writing statistical commentary"
},
{
- "objectID": "CODE_OF_CONDUCT.html#enforcement",
- "href": "CODE_OF_CONDUCT.html#enforcement",
- "title": "Contributor Covenant Code of Conduct",
- "section": "",
- "text": "Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at statistics.development@education.gov.uk. All complaints will be reviewed and investigated promptly and fairly.\nAll community leaders are obligated to respect the privacy and security of the reporter of any incident."
+ "objectID": "writing-visualising/cd.html#headline-sections",
+ "href": "writing-visualising/cd.html#headline-sections",
+ "title": "Writing and visualising",
+ "section": "Headline sections",
+ "text": "Headline sections\nIt is common to use bullet points to draw out key headline messages, either for policy lines or for general interest. Here are some top tips for writing headlines:\n\nAsk yourself if this is the most important, useful and relevant point to make? Why? What is new?\nWe recommend a maximum of six top headlines\nYou shouldn’t be trying to summarise all the findings in the publication\nFor regular publications, headlines won’t necessarily be the same every time\nHeadlines should be a single sentence making a single point, and be able to stand alone from the publication\nHeadlines should make sense to everyone and anyone (no jargon)\nStructure headlines as: what has happened; why is this important - don’t give numbers without context\nAt least of your headlines should put the latest figures in the context of the longer-term change\nRound figures in headlines, you don’t need lots of decimal places\nIf there is essential context for the headline facts then put this here"
},
{
- "objectID": "CODE_OF_CONDUCT.html#enforcement-guidelines",
- "href": "CODE_OF_CONDUCT.html#enforcement-guidelines",
- "title": "Contributor Covenant Code of Conduct",
- "section": "",
- "text": "Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:\n\n\nCommunity Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.\nConsequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.\n\n\n\nCommunity Impact: A violation through a single incident or series of actions.\nConsequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.\n\n\n\nCommunity Impact: A serious violation of community standards, including sustained inappropriate behavior.\nConsequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.\n\n\n\nCommunity Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.\nConsequence: A permanent ban from any sort of public interaction within the community."
+ "objectID": "writing-visualising/cd.html#content-headings",
+ "href": "writing-visualising/cd.html#content-headings",
+ "title": "Writing and visualising",
+ "section": "Content headings",
+ "text": "Content headings\nThroughout the commentary, active sub-headings outline the main messages making them memorable for users. They should be a short summary of the contents - aim for 10 words or less - rather than a full statistical headline. Avoid using exaggerated or sensationalist language that you would not use in your main commentary such as “highest ever” or “only”. Be neutral in selecting the message to use in the heading rather than favouring positive or negative messages."
},
{
- "objectID": "CODE_OF_CONDUCT.html#attribution",
- "href": "CODE_OF_CONDUCT.html#attribution",
- "title": "Contributor Covenant Code of Conduct",
- "section": "",
- "text": "This Code of Conduct is adapted from the Contributor Covenant, version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.\nCommunity Impact Guidelines were inspired by Mozilla’s code of conduct enforcement ladder.\nFor answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations."
+ "objectID": "writing-visualising/cd.html#accordion-section-content",
+ "href": "writing-visualising/cd.html#accordion-section-content",
+ "title": "Writing and visualising",
+ "section": "Accordion section content",
+ "text": "Accordion section content\nYou should split your release into sections that each focus on one or two key messages, with a recommended maximum of 10 sections in the release. The whole release should take no more than 10 minutes to read. Our analytics app contains insights on how long it takes the average user to read your release.\nTo keep the release short only include information if there is something interesting to say - the commentary is there to tell a story, people looking for specific figures will use the table tool, or download the underlying data instead. Do not try to summarise every number in the commentary.\nAvoid having large blocks of text as they are hard to read and users scan them and miss the detail. Graphs and tables break up the content but only include these where they add value; you do not need a graph or chart in every section.\nUse plain English and shorter sentences, aim for an average of 15-20 words per sentence. Do not overload sentences with numbers and avoid ‘elevator commentary’ that describes small movements in the whole series without giving any insight (use a summary table instead if it is interesting, or leave it out entirely). Be impartial and objective; avoid using sensationalist terms or terms that reflect a judgement such as “very few” “only” or “strong”.\nExplain complex concepts in plain English within the text. Remember that for many of our users, confidence intervals and significant differences are complex concepts that need explaining.\n\nDo not use footnotes in the text of your content. They’re designed for reference in print, not web pages. Always consider the user need first. If the information in the footnotes is important, include it in the body text. If it’s not, leave it out."
},
{
- "objectID": "index.html",
- "href": "index.html",
- "title": "Analysts’s Guide",
- "section": "",
- "text": "This website is a guide aimed at anyone working in analysis or statistics in the Department for Education (DfE). It includes tips on best practice and learning resources across a number of areas.\nWe hope it can prove a useful community driven resource for everyone from the most experienced analyst right through to those just starting out. If you have any feedback, suggested additions, or wish to challenge any of the guidance, feel free to use the GitHub links to suggest changes directly, or contact us at the email you can find at the bottom of the page."
+ "objectID": "writing-visualising/cd.html#writing-about-characteristics",
+ "href": "writing-visualising/cd.html#writing-about-characteristics",
+ "title": "Writing and visualising",
+ "section": "Writing about characteristics",
+ "text": "Writing about characteristics\nThere is a wide range of guidance available from the GSS, ONS and the Cabinet Office around writing about characteristics. Statistics content published on EES should adhere to the principles outlined by the above. The data harmonisation champions group are in the process of collating the most recent guidance from all these sources and summarising it below.\nIf you need some steer on how to report on a particular characteristics, the below links provide some useful starting points:\n\nAnalysis Function Data Harmonisation\nCabinet Office guidance on writing about statistics\n\nYou can also get in touch with the DfE Data Harmonisation Champions Group via statistics.development@education.gov.uk.\n\n\nWriting about ethnicity\n\nFor the official names of ethnicity filters to use in data files, please check our guidance on creating statistics. The below outlines some key points in writing about ethnicity in publication content.\n\n\nEthnic minorities and not BAME\n\nAs outlined in the CRED report, grouping ethnicity at a higher level than the 5 major ethnic groups should be avoided as it risks masking the true disparities between, and/or life experiences of, people from different ethnic groups.\nThere will, however, be specific cases where data quality or comparisons with past metrics does not allow for statistics to be produced at the level of the major or minor ethnic groups. In these cases, the group ‘ethnic minorities’ should be used exclusively to refer to all ethnic groups except the White British group, this includes White minorities, such as Gypsy, Roma and Irish Traveller groups.\nFor comparisons with the White group as a whole, ‘all other ethnic groups combined’ or ‘ethnic minorities (excluding White minorities)’ should be used. It may also be necessary to refer to ‘White’ and ‘other than White’ if space is limited, but this should be avoided if possible.\nDo not use the terms ‘non-White’, BAME (Black, Asian, and Minority Ethnic), or BME (Black and minority ethnic). These terms can be perceived as demeaning and can be confusing as to what groups are being included and excluded. For comparison with past metrics, BAME and BME should be replaced by the term “ethnic minorities (excluding White minorities)” unless White minorities were specifically mentioned.\n\n\n\nOrdering of groups\n\nThere are several methods of ordering the ethnic groups, depending on where and how they are used:\n\nAlphabetical: use in tables and when listing ethnic groups (with ‘Other’ and sometimes ‘Unknown’ as a final category)\nIn expected order of size (with largest first): useful in charts and visualisations as it makes data and patterns easier to read"
},
{
- "objectID": "index.html#learning-and-development",
- "href": "index.html#learning-and-development",
- "title": "Analysts’s Guide",
- "section": "Learning and development",
- "text": "Learning and development\nGeneral resources - Useful learning resources, and support to get you started\nSQL - Guidance and tips for accessing data via databases with SQL\nR - Guidance and tips for using R\nGit - Guidance and tips for version control with Git\nPython - Guidance and tips for using Python [NEW PAGE]"
+ "objectID": "writing-visualising/cd.html#reporting-and-interpreting-numbers",
+ "href": "writing-visualising/cd.html#reporting-and-interpreting-numbers",
+ "title": "Writing and visualising",
+ "section": "Reporting and interpreting numbers",
+ "text": "Reporting and interpreting numbers\nReporting the numbers you work with is an important first step toward writing effective numeric descriptions. By including numbers in text, table or chart, you give the user the raw materials with which to perform comparisons across time, places or groups. However, if you stop there, you leave it to your readers to figure out how those data answer the question at hand.\nPoor: “In 2010, there were 20, 370 overall homicides related to crime, 13,000 which were related to gun incidents, 7,370 related to other weapons. In 2020, they were 18,900 overall homicides, 11,000 which were related to gun incidents, 9,900 related to other weapons (Figure 1)”.\nThe description above simply lists statistics from charts without explaining how they relate to one another or how the statistics address the initial question in the opening paragraph.\nBetter: “The total number of homicides rose until the mid-2000s and then declined until 2020. As shown in Figure 1, the increase and subsequent decrease in homicides were driven by trends in gun-related homicides. In 2020, there was roughly 1.5 times as many homicides were committed with guns as with other types of weapons (11,000 versus 7265; Figure 1); whereas in 2010, roughly 2 times as many homicides were committed with guns versus other weapons, 13000 and 6500, respectively. Figure 2 examines whether gun-related homicides showed the same time trend among all age groups. As shown in Figure 2 in the two youngest groups of offenders, gun-related homicides increased substantially between 2000 and 2010, and then decreased steadily until 2020. In contrast, the number of gun related homicides committed by older offenders decreased slowly throughout the time-period shown”.\nTry to use prose to summarise the patterns so your user can see the general relationship in the table or chart – the forest not the individual trees. Try not to report every number from the table or chart or pick a few arbitrary numbers to contrast in sentence form without considering whether or not those numbers represent an underlying general pattern. Paint the big picture rather than reiterating all the little details. This will help you tell a clear story with numbers as evidence."
},
{
- "objectID": "index.html#statistics-production",
- "href": "index.html#statistics-production",
- "title": "Analysts’s Guide",
- "section": "Statistics production",
- "text": "Statistics production\nHow to publish - Guidance for how to publish different types of statistics\nOpen data standards - Guidance on how to structure data files\nPublication pipelines - Link to the RAP for stats? [NEW PAGE / REDIRECT]\nUsing EES - How to use the features in Explore Education Statistics\nGood examples in EES - Examples of good practice in Explore Education Statistics\nEmbedded visualisations in EES - How to embed R-Shiny charts in EES publications\nUser engagement - Guidance on understanding and engaging with the users of published statistics\nEES analytics - Understanding how users are interacting with your publications"
+ "objectID": "writing-visualising/cd.html#creating-hyperlinks",
+ "href": "writing-visualising/cd.html#creating-hyperlinks",
+ "title": "Writing and visualising",
+ "section": "Creating hyperlinks",
+ "text": "Creating hyperlinks\nAvoid using full URLs in text. Hyperlinks should be used and they should provide a clear description of the destination. Avoid using ‘For more information click here’. Screen readers often collate all links on a page into one list, so having numerous ‘click here’ links listed is confusing to the user and gives no description of the destinations. ‘For more information see Guidance to support the summer 2021 exams’ is an example of a good hyperlink.\nBest practice on creating hyperlinks, particularly how to name them and common pitfalls to avoid can be found on this introduction to html guide. Those of us using EES don’t need to worry about writing the raw html for the anchor links, and should instead focus on the section referring to how to name and title links."
},
{
- "objectID": "index.html#writing-and-visualising-data",
- "href": "index.html#writing-and-visualising-data",
- "title": "Analysts’s Guide",
- "section": "Writing and visualising data",
- "text": "Writing and visualising data\nPublic dashboards - Guidance for publishing public facing statistics dashboards\nWriting about statistics - Things to consider when writing about statistics and data\nVisualising data - Resources and best practice to guide you when visualising data [NEW PAGE]"
+ "objectID": "writing-visualising/cd.html#search-engine-optimisation",
+ "href": "writing-visualising/cd.html#search-engine-optimisation",
+ "title": "Writing and visualising",
+ "section": "Search engine optimisation",
+ "text": "Search engine optimisation\nSearch engine optimisation (SEO) makes it easier for users to find your data through search engines like Google. Some top tips include:\n\nKeeping your release title shorter than 50-60 characters. This means the full title can be displayed on the search engine results\nAvoid listing key words: search engines penalise anything not recognised as a full sentence.\nMake use of our analytics app to explore what your users are doing: what accordions are they clicking on, what are they searching for? This could give an idea of what content you should focus on in future, and which areas are no longer of interest to most of your users.\n\nFollowing best practice in writing about statistics is of increasing importance. As shown in the below example any sentence could be pulled out into a snippet and shown in a search engine to users who are searching for related information:\n\nWe should make a concerted effort to ensure that we are answering the questions people are interested in as search engines are getting smarter and pulling this information directly out of webpages. See the following example of a google search using a snippet from one of our publications as an answer in the search engine results itself:"
},
{
- "objectID": "index.html#reproducible-analytical-pipelines-rap",
- "href": "index.html#reproducible-analytical-pipelines-rap",
- "title": "Analysts’s Guide",
- "section": "Reproducible Analytical Pipelines (RAP)",
- "text": "Reproducible Analytical Pipelines (RAP)\nRAP expectations - Guidance for all analysts on expectations of RAP\nRAP support - Details on support available for RAP in DfE\nRAP FAQs - Frequently asked questions about RAP\nRAP in statistics - Detailed RAP guidance for statistics publications"
+ "objectID": "writing-visualising/cd.html#visualisation-titles",
+ "href": "writing-visualising/cd.html#visualisation-titles",
+ "title": "Writing and visualising",
+ "section": "Visualisation titles",
+ "text": "Visualisation titles\nFor tables and charts, active titles are descriptive and tell the trend by highlighting the main story. They should be short - aim for 10 words or less and avoid going over more than one line. Active titles give users the main message without having to find the text that accompanies the chart, and makes it easier for journalists to use your chart directly without having to write their own summary.\nPut information such as the measure, source, population, geographical coverage and time period in a caption if they are not obvious from the chart content. Add further context and information in the main text. Remember that tables and charts should be usable even if isolated from the rest of the release."
},
{
- "objectID": "index.html#ada-and-databricks",
- "href": "index.html#ada-and-databricks",
- "title": "Analysts’s Guide",
- "section": "ADA and databricks",
- "text": "ADA and databricks\nSomething - ? TBC"
+ "objectID": "writing-visualising/cd.html#using-data-blocks-effectively",
+ "href": "writing-visualising/cd.html#using-data-blocks-effectively",
+ "title": "Writing and visualising",
+ "section": "Using data blocks effectively",
+ "text": "Using data blocks effectively\nAim for fewer tables, and keep them small. As a guide, we suggest no more than one table per accordion section.\nPresentation tables are the tables you include within the accordion sections of your release to quickly visualise numbers. Unlike the underlying data files, the presentation tables focus on specific parts of the story you are telling. They are distinct from, and should never be a copy of an underlying data file.\nYou should only be including small tables sparingly where they add value to your release and not use them as a straight copy of the ready-made Excel tables previously published on gov.uk. The data itself is there for users to access through other means, therefore any data in the commentary should only be there if it is enhancing the story.\n\nYou do not need to recreate all of the old excel tables, users can find the numbers they are interested in using the table tool, or analyse the underlying data if they want the data behind the release."
},
{
- "objectID": "index.html#contact-us",
- "href": "index.html#contact-us",
- "title": "Analysts’s Guide",
- "section": "Contact us",
- "text": "Contact us\nOur mailbox is always monitored and is available for anyone in DfE to ask questions about statistics, whether that is about RAP, building dashboards, coding support, learning and development or statistics publications.\n\nstatistics.development@education.gov.uk\n9am-5pm, Monday-Friday, aim to reply within 1-2 days"
+ "objectID": "writing-visualising/cd.html#charts-in-ees",
+ "href": "writing-visualising/cd.html#charts-in-ees",
+ "title": "Writing and visualising",
+ "section": "Charts in EES",
+ "text": "Charts in EES\nAny charts in your release should be appropriate for the data, reinforcing the key messages. For more detailed information on how to build charts in EES, visit our EES charts guidance section.\nAny analyst building charts in or outside of EES should make themselves familiar with the [GSS data visualisation guidance](GSS colours in visualisations guidance.\nMore complex charts cannot currently be built in EES. You should only use complex charts where there is a clear user need, as simple charts are the easiest for users to understand. Please see the ONS guidance for visualisations or contact the Statstics Development Team if you’re considering a more complex chart built outside of EES.\nDWP have created a Data Visualisation Thinking course that may be useful to look at when creating more complex charts."
+ },
+ {
+ "objectID": "writing-visualising/cd.html#colours-in-charts",
+ "href": "writing-visualising/cd.html#colours-in-charts",
+ "title": "Writing and visualising",
+ "section": "Colours in charts",
+ "text": "Colours in charts\nThe most important consideration when using colour is to avoid relying on it for interpretation. It should be seen as an enhancement, and your charts should be understandable without it.\nFor any charts made outside of EES you should use the suggested colours from the GSS colours in visualisations guidance. The following sections show the colours that are available for charts in EES and suggest the way to use them with one another, more detail on why can be found in the GSS guidance itself.\nWe strongly recommend you stick to the codes as outlined, if you can’t for any reason, please make sure you follow the guidance on developing your own colour palette and get in touch with us so we can understand why.\n\n\nCategorical colour palette\n\nCategorical data can be divided into groups or categories by using names or labels. This palette has four colours. We recommend a limit of four categories as best practice for basic data visualisations. The ordering of the palette is important as not all colours are accessible when paired together.\n\n\n\nColour name\nHex code\nRGB\nCYMK\n\n\n\n\nDark blue\n#12436D\n18, 67, 109\n36, 16, 0, 57\n\n\nTurquoise\n#28A197\n40, 161, 151\n75, 0, 6, 37\n\n\nDark pink\n#801650\n128, 22, 80\n0, 83, 38, 50\n\n\nOrange\n#F46A25\n244, 106, 37\n0, 57, 85, 4\n\n\n\n\n\n\nSequential colour palette\n\nSequential data is any sort of data where the order of series has some meaning. For example, age groups ascending in age. This palette has three colours and even this pushes the boundaries for contrast ratios. Any charts made with shades of a specific hue should be accessible without colour.\n\n\n\nColour name\nHex code\nRGB\nCYMK\n\n\n\n\nDark blue\n#12436D\n18, 67, 109\n36, 16, 0, 57\n\n\nMid blue\n#2073BC\n32, 115, 188\n0, 57, 85, 4\n\n\nLight blue\n#6BACE6\n107, 172, 230\n75, 0, 6, 37\n\n\n\n\n\n\nFocus palette\n\nOn focus charts, colour is used to highlight specific elements to help users understand the information. One line out of many will be highlighted as a colour and the rest of the series remain grey.\n\n\n\nColour name\nHex code\nRGB\nCYMK\n\n\n\n\nDark blue\n#12436D\n18, 67, 109\n36, 16, 0, 57\n\n\nGrey\n#BFBFBF\n191, 191, 191\n0, 0, 0, 25\n\n\n\n\n\n\nLarger palettes\n\nAll charts should be made as simple as possible so that the message is easy to understand and interpret.\nIf you do want to use additional colours you need to ensure that the chart is understandable without colour as no series with more than four colours will ever be fully accessible alone.\nFor categorical data, we recommend the following extensions to the categorical palette. Once again, the ordering is important. If you’re making a chart in EES these are not default colours, so you will need to use the colour picker to specify them.\n\n\n\nColour name\nHex code\nRGB\nCYMK\n\n\n\n\nDark blue\n#12436D\n18, 67, 109\n36, 16, 0, 57\n\n\nTurquoise\n#28A197\n40, 161, 151\n75, 0, 6, 37\n\n\nDark pink\n#801650\n128, 22, 80\n0, 83, 38, 50\n\n\nOrange\n#F46A25\n244, 106, 37\n0, 57, 85, 4\n\n\nDark grey\n#3D3D3D\n61, 61, 61\n0, 0, 0, 76\n\n\nLight purple\n#A285D1\n162, 133, 209\n22, 36, 0, 18\n\n\n\nFor larger palettes when using sequential data, we recommend using ColorBrewer to find the hex codes for multiple shades of a given hue."
},
{
- "objectID": "learning-development/learning-support.html",
- "href": "learning-development/learning-support.html",
- "title": "Learning support",
- "section": "",
- "text": "A collection of useful learning resources, and information on support and resources to get you started\nAs statistics producers we require a variety of tools to be able to process our data. Below are the recommended tools that will give us the most power to do what we need. These have large user communities in DfE, and are already working in our current IT setup.\nFor best practice when using software and coding in our process, see our guidance on RAP expectations the DfE Good Code Practice guide."
+ "objectID": "writing-visualising/cd.html#visualising-breaks-in-a-timeseries",
+ "href": "writing-visualising/cd.html#visualising-breaks-in-a-timeseries",
+ "title": "Writing and visualising",
+ "section": "Visualising breaks in a timeseries",
+ "text": "Visualising breaks in a timeseries\nWe recommend including any missing data from breaks in a time series in your data file using the appropriate GSS symbol, such as in this example table:\n\n\n\n\n\n\n\n\n\n\n\n\n2013/14\n2014/15\n2015/16\n2016/17\n2017/18\n\n\n\n\nNumber of pupil enrollments\n3,627,985\n3,713,774\n3,796,146\nx\n3,885,774\n\n\nNumber of schools\n16,705\n16,723\n16,736\nx\n16,739\n\n\n\n\nThere may be times when including missing data increases the file size too much, or becomes unwieldy, if you’re unsure and would like advice on your data contact statistics.development@education.gov.uk.\n\nBy including the missing data in your open data files you can then create charts in EES that represent this. Start off by creating a data block with the data you want to build the chart from.\n\nOn the chart configuration tab there is a toggle for visualising non-numeric values in the data as 0. On the first chart configuration tab, there’s a check box that will toggle between showing and hiding them.\n\nWhen selected, you will then see that this data appears in the chart as if the indicator values are 0."
},
{
- "objectID": "learning-development/learning-support.html#email-support",
- "href": "learning-development/learning-support.html#email-support",
- "title": "Learning support",
- "section": "Email support",
- "text": "Email support\n\nstatistics.development@education.gov.uk\n\n9-5, aim to reply between 1-2 days."
+ "objectID": "writing-visualising/cd.html#glossary",
+ "href": "writing-visualising/cd.html#glossary",
+ "title": "Writing and visualising",
+ "section": "Glossary",
+ "text": "Glossary\nThe glossary in Explore Education Statistics is intended for definitions used across all publications. Please send any definitions you would like adding to the glossary to the Statistics Development Team.\nOnce definitions have been added, you can add them as URL links into your commentary. E.g. to see the definition for ‘Otherwise vulnerable’, you’d make that phrase a hyperlink, and use https://explore-education-statistics.service.gov.uk/glossary#otherwise-vulnerable as the link.\nBefore creating a new definition, check the glossary to see if one is already there. When creating glossary entries you should use standard DfE definitions, only write your own if it is something that relates specifically to your statistics. Do not use technical terms in your definition, stick to plain English.\nInclude essential definitions in the commentary where they are relevant, you should not expect users to go to the glossary or about these statistics section for information they need to understand the statistics. Do not use the glossary to explain methodology such as classification decisions, explain these in the text if essential or otherwise in the methodology document."
},
{
- "objectID": "learning-development/learning-support.html#technical-workshops",
- "href": "learning-development/learning-support.html#technical-workshops",
- "title": "Learning support",
- "section": "Technical workshops",
- "text": "Technical workshops\n\nIn person workshops covering specific technical skills in practice\n3 hours long, with people working in small groups\nRecently circled the sites running an introduction to Git, and introduction to R and RAP.\nContact statistics.development@education.gov.uk to register interest in workshops happening at your site or to request new topics!"
+ "objectID": "writing-visualising/cd.html#publication-scrums",
+ "href": "writing-visualising/cd.html#publication-scrums",
+ "title": "Writing and visualising",
+ "section": "Publication scrums",
+ "text": "Publication scrums\nThe Statistics Services Unit organise regular publication scrums, which enable publications to be looked at with fresh eyes through a workshop with volunteers and the publication team. The content checklist, which breaks down the components of producing Best Practice content into smaller chunks with prompt questions is used (Download content checklist), along with workshop slides.\nIf you are responsible for signing off publications, then please download and see the Statistics Leadership Group paper highlighting top tips for content design.\nOne of the early outcomes from running publication scrums, was that great content design is more of an art form than science and the scrum process was found to be really useful in enabling a wider element of scrutiny and friendly challenge, in an enjoyable environment to consider publication content.\nYou can experience how a scrum runs through watching this scrum-along (features the scrum up until groups breakout to discuss different elements) and supporting slides\n\n\n\n\nA full before / after scrum along is available to watch, with slides available to download separately.\nFor a walk through of some of the end to end benefits the scrums have had, take a look at this video for Schools and pupils Statistics Team.\nFinally, an example of a team who has been through the scrum process for 3 publications talk through their realised benefits, showcasing the type of benefits potentially others could also realise is also available to watch on sharepoint.\nIf you’re interested in being involved in future scrums, please get in contact us."
},
{
- "objectID": "learning-development/learning-support.html#partnership-programme",
- "href": "learning-development/learning-support.html#partnership-programme",
- "title": "Learning support",
- "section": "Partnership programme",
- "text": "Partnership programme\n\nDedicated resource to support development work, either building skills or working on statistics outputs.\nA short-term project with an agreed set of goals and timescale decided as part of an initiation meeting.\nContact statistics.development@education.gov.uk if you’re interested in finding out more about the support we can give."
+ "objectID": "writing-visualising/dashboards.html",
+ "href": "writing-visualising/dashboards.html",
+ "title": "Public dashboards",
+ "section": "",
+ "text": "Guidance for publishing public statistics dashboards"
},
{
- "objectID": "learning-development/learning-support.html#specific-resources",
- "href": "learning-development/learning-support.html#specific-resources",
- "title": "Learning support",
- "section": "Specific resources",
- "text": "Specific resources\nWe also have specific learning resources and materials for SQL, R, and Git"
+ "objectID": "writing-visualising/dashboards.html#user-needs",
+ "href": "writing-visualising/dashboards.html#user-needs",
+ "title": "Public dashboards",
+ "section": "User needs",
+ "text": "User needs\nWhen designing a government service, always start by learning about the people who will use it. If you do not understand who they are or what they need from your service, you cannot build the right thing.\nUnderstanding as much of the context as possible gives you the best chance of meeting users’ needs in a simple and cost effective way.\nThe real problem might not be the one you originally thought needed solving. Testing your assumptions early and often reduces the risk of building the wrong thing.\nServices designed around users and their needs:\n\nare more likely to be used\nhelp more people get the right outcome for them - and so achieve their policy intent\ncost less to operate by reducing time and money spent on resolving problems\n\nSee the service manual for more information on learning about your users and their needs."
},
{
- "objectID": "learning-development/r.html",
- "href": "learning-development/r.html",
- "title": "R",
- "section": "",
- "text": "Guidance and tips for using the programming language R"
+ "objectID": "writing-visualising/dashboards.html#software-choices",
+ "href": "writing-visualising/dashboards.html#software-choices",
+ "title": "Public dashboards",
+ "section": "Software choices",
+ "text": "Software choices\nWhen you make a decision about technology, you’re making a significant investment. The choices you make will have a huge impact on your ability to create, iterate and operate the service in a sustainable way. We should be choosing tools and technology that let us create a high quality service in a cost effective way and minimise the cost of changing direction in future.\nFor public facing dashboards we recommended using R Shiny.\nWe feel that it best meets the service standards, it aligns with the departments RAP strategy for the use of R in in Official Statistics, and the general direction of the government analytical community.\nR Shiny is incredibly customisable, and ideal for providing everything in one place - interactivity, user customised printable PDFs (replacing old mail merge solutions), and ease of use on a phone!\nR Shiny takes advantage of transferable knowledge and shared resources that already exist to minimise the costs and maximise the shared benefits.\nWhile we recommend R Shiny, teams may use other tools if it can be justified against the Service Standard 11. Choose the right tools and technology.\nThe most common dashboard options available to us are:\n\nR Shiny\nPython Dash (or other Python-based alternatives)\nPowerBI\n\nSome considerations to think about when choosing the tooling / software for a dashboard:\n\nwill it be open source? (Service standards 12 and 13)\nwhat skills do your team already have, or are already developing?\ncan you reuse anything that someone else has already done?\ndo you have flexibility in formatting and styling?\nwill you be able to maintain it long term?\ndoes it give the flexibility required to meet accessibility requirements?\nwhat costs will it involve? (Consider learning and development time and courses as well as hosting)\nwill it allow you to develop automated testing and CI to QA our dashboards?\ndoes it align with the AF expectation to move towards R and Python for analysis?\n\nThe guidance on this page will focus on R Shiny. However, if you are using PowerBI you should also make use of the department’s PowerBI Dashboard standards, and aim to make use of reusable configurations where possible to ensure that duplication of effort across the department is minimised."
},
{
- "objectID": "learning-development/r.html#what-is-r",
- "href": "learning-development/r.html#what-is-r",
- "title": "R",
- "section": "What is R",
- "text": "What is R\nR is an open-source programming language specifically aimed at statisticians and data analysts."
+ "objectID": "writing-visualising/dashboards.html#accessibility",
+ "href": "writing-visualising/dashboards.html#accessibility",
+ "title": "Public dashboards",
+ "section": "Accessibility",
+ "text": "Accessibility\nGovernment services must work for everyone who needs to use them. Public sector organisations have a legal duty to consider everyone’s needs when they’re designing and delivering services. Making a website or mobile app accessible means making sure it can be used by as many people as possible. This includes those with:\n\nimpaired vision\nmotor difficulties\ncognitive impairments or learning disabilities\ndeafness or impaired hearing\n\nAccessibility means more than putting things online. It means making your content and design clear and simple enough so that most people can use it without needing to adapt it, while supporting those who do need to adapt things.\nThe accessibility regulations came into force for public sector bodies on 23 September 2018. They say you must make your website or mobile app more accessible by making it ‘perceivable, operable, understandable and robust’. The full name of the accessibility regulations is the Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018. The accessibility regulations build on your existing obligations to people who have a disability under the Equality Act 2010 (or the Disability Discrimination Act 1995 in Northern Ireland).\nThese regulations apply to all dashboards created for Official statistics. All dashboards should:\n\nhave an accessibility statement\nfollow the guidance below for testing against WCAG 2.1 AA accessibility standard\nuse open data that is already available via EES\n\nAnalysts should familiarise themselves with the gov.uk guidance on service accessibility"
},
{
- "objectID": "learning-development/r.html#what-is-r-for",
- "href": "learning-development/r.html#what-is-r-for",
- "title": "R",
- "section": "What is R for",
- "text": "What is R for\nR can be used for almost anything you can think of, notably data analysis, data visualisation, and creating reports and dashboards. It can also be used to extract data from SQL databases and run SQL queries."
+ "objectID": "writing-visualising/dashboards.html#governance",
+ "href": "writing-visualising/dashboards.html#governance",
+ "title": "Public dashboards",
+ "section": "Governance",
+ "text": "Governance\nAll teams developing dashboards to accompany Official Statistics should contact the Statistics Development Team for advice on the level of approval required as there may be times when HoP approval is needed in addition to team/unit leader approval. Approvals should be sent to the Statistics Development Team when asking for a new dashboard to be hosted on shinyapps.io.\nThe standards for dashboards, and department strategy are maintained by the Central Statistics Standards Unit and governed by the Statistics Leadership Group, which is made up of all senior statisticians owning Official Statistics publications in the department.\nRegular reflection on how teams are doing and finding ways to improve is an important part of good governance. We recommend that Senior Statisticians responsible for dashboards ensure that they are regularly reviewed, covering:\n\ntesting and quality assurance\naccessibility\nperformance against user needs\noverall coherence with central standards and strategy found on this page"
},
{
- "objectID": "learning-development/r.html#how-to-install-r",
- "href": "learning-development/r.html#how-to-install-r",
- "title": "R",
- "section": "How to install R",
- "text": "How to install R\nDownload R (language) and RStudio (IDE) from the DfE software center. We also recommend that you download RTools (a helpful R extension) at the same time.\nThere are usually a couple of different versions available for software on the software center, we’d recommend you always go for the latest (newest) version possible."
+ "objectID": "writing-visualising/dashboards.html#iterative-development",
+ "href": "writing-visualising/dashboards.html#iterative-development",
+ "title": "Public dashboards",
+ "section": "Iterative development",
+ "text": "Iterative development\nUsing agile methods means getting your service in front of real users as soon as possible. Then observing and generating data on how they use it and iterating the service based on what you’ve learned. Because you’re not specifying everything up front before you’ve developed an understanding of what users need, agile methods reduce the risk of delivering the wrong thing.\nTeams should aim to get a working version of their dashboard out to users as soon as possible to prototype the design and content.\nIteration isn’t just for the early stages of a service’s development and services are never ‘finished’. Using agile methods means getting real people using your service as early as possible. Then making improvements throughout the lifetime of the service.\nMaking improvements means more than doing basic maintenance like fixing bugs in code, deploying security patches and keeping call centre scripts up to date. If that’s all you do, you’ll be fixing symptoms rather than underlying problems. And over time, the service will stop meeting user needs.\nContinuous improvement means you can respond to changes in user needs, technology or government policy throughout the lifetime of the service. So rather than having to be replaced, the service stays relevant until it’s ready to be retired."
},
{
- "objectID": "learning-development/r.html#best-places-to-start",
- "href": "learning-development/r.html#best-places-to-start",
- "title": "R",
- "section": "Best places to start",
- "text": "Best places to start\n\nThe DfE Analytics Academy host an online R training course. This is a great resource full of reproducible examples using DfE data. The course takes you through initially getting R downloaded, all the way through to developing apps in RShiny.\nThere is also the DfE R training guide, which is a great starting point and reference to guide you through how to get started using R and RStudio.\nAs an alternative, with a number of options for beginners to R, RStudio Education provide a variety of materials to suit different learning styles."
+ "objectID": "writing-visualising/dashboards.html#code-testing",
+ "href": "writing-visualising/dashboards.html#code-testing",
+ "title": "Public dashboards",
+ "section": "Code testing",
+ "text": "Code testing\nTesting is a way to capture desired behaviour of your code, in such a way that you can automatically verify that it keeps working the way you expect over time. It is essential for making sure that code works the way that you intend it to, and keeps working even after you make changes to the code so that your users have access to a stable service. You need to test your service regularly as part of quality assurance (QA) to make sure that it:\n\nis easy to use for anyone who needs to use it, regardless of the device they’re using\nis stable, secure and works quickly, regardless of how many people need to use it\ncan be iterated quickly to meet changes to user needs or the political environment\n\nTests can come in a variety of shapes and sizes, good starting points for analysts new to testing are the Duck Book, and DfE good code practice.\nYou should aim to automate as much of your testing as possible and run your test suite as part of continuous integration (where your tests form part of your codebase). By testing your code automatically every time you make a change, you’ll be able to find defects more quickly. Getting fast feedback means you can respond to any problems quickly and make changes when you need to. You can also spot defects before they develop into bigger problems that are more complicated and expensive to fix.\nFor dashboards created in R Shiny, we strongly recommend a mix of unit tests and UI tests (integration), run using GitHub Actions for continuous integration (CI). All applications should have an appropriate level of test coverage before being published. More information and details on how to get started with this can be found in the testing R Shiny section below."
},
{
- "objectID": "learning-development/r.html#best-practice",
- "href": "learning-development/r.html#best-practice",
- "title": "R",
- "section": "Best practice",
- "text": "Best practice\nTips for reaching best practice in R can be found on our RAP page, with guidance on meeting best practice in RAP for clean final code. This makes it easier to read and pick up if another person is running your code."
+ "objectID": "writing-visualising/dashboards.html#assessing-engagement",
+ "href": "writing-visualising/dashboards.html#assessing-engagement",
+ "title": "Public dashboards",
+ "section": "Assessing engagement",
+ "text": "Assessing engagement\nWork out what success looks like for your service and identify metrics which will tell you what’s working and what can be improved, combined with user research. For dashboards this will likely be things like the number of users and interactions with what you have created.\nDefining what “good” looks like and identifying appropriate metrics means that you’ll know whether the service is solving the problem that it’s meant to solve.\nCollecting the right engagement data means you’ll be alerted to potential problems with your service. And when you make a change to the service, you’ll be able to tell whether it had the effect you expected. In practice this will often mean setting up analytics and feedback surveys for dashboards and monitoring the data you get back.\nAt a minumum you should be requesting feedback from users via a survey hosted on the dashboard and reviewing this on a regular basis. An example of this kind of feedback survey is the beta banner survey on Explore Education Statistics.\n\nGoogle Analytics is a free service that collects information on who visits your webpage and how they interact with it. You can set up basic Google Analytics for your published dashboard in a few simple steps outlined in this article: Add Google Analytics to a Shiny app and view a more complex example in this example file. If you’re planning to publish a dashboard, or to set up Google Analytics for a published dashboard, please contact statistics.development@education.gov.uk."
},
{
- "objectID": "learning-development/r.html#how-to-work-with-r",
- "href": "learning-development/r.html#how-to-work-with-r",
- "title": "R",
- "section": "How to work with R",
- "text": "How to work with R\n\nR Projects\n\nWhenever you are using R, you should work in an RProject. This just makes sure you are set up in the correct working directory, so your code is pointing at the right folders and files.\nThis guide for using projects in R is a really useful article to help you set up a project.\nYou can check which project you are working in by looking in the top right hand corner of RStudio:\n\n\n\n\nOutlines\n\nIn RStudio you can greatly increase the navigability of your code by taking advantage of outlines. More information on folding and navigating outlines in RStudio can be found online, though when using rmarkdown reports, remember to use names first, such as ## Rows that aren't matching: r nrow(joined %>% filter(matching == FALSE)), rather than having the R code first, so that they are easy to discern in the outline.\n\n\n\nrenv\n\nYou should use the renv package for package and version control in R.\nPackages and versions of R regularly update. Over time, this can cause code to break - e.g. if different dependencies are required for later versions of packages to work. Using renv creates a “snapshot” of your code and packages at the time you created it, which anyone can then recreate when they come to use your code.\nThis is really important for reproducibility, and will help you meet elements of great practice with recyclable code for future use.\n\n\nrenv::restore()\n\nSometimes renv::restore() can fail, and when in specific renv-controlled projects install.packages() will fail saying that packages aren’t available even when they clearly are. There are a couple of workarounds we have found that get around this failure.\n\nConfiguring the proxy settings by running the below in R - this also helps if you are getting timeout issues when trying to webscrape with R:\n\nSys.setenv(no_proxy=\"*\") \n\n\nSpecifying the renv library as the install location. It’s a bit of a fudge, though these lines are helpful to get the packages from the renv lockfile installed and you running the project when needed:\n\nmyPath <- .libPaths()[1]\n\nforceInstall <- function(pkg, path) {\nmissing <- suppressWarnings(eval(parse(text= paste0(\"!require(\",pkg,\")\"))))\n\nif(missing == FALSE){\nmessage(pkg, \" is already installed.\")\n} else{\ninstall.packages(pkg, lib = path)\n}\n}\n\nforceInstall(\"jsonlite\", myPath)\n\nrenvPackages <- names(jsonlite::fromJSON(\"renv.lock\", flatten = TRUE)$Packages)\n\ninvisible(lapply(renvPackages, forceInstall, path = myPath))\nMore manual equivalent to use for specific packages:\n.libPaths() # note down output 1, and reuse in the lib argument of install.packages() as below\n\ninstall.packages(\"rmarkdown\", lib = \"C:/Users/swong/OneDrive - Department for Education/Documents/stats-production-guidance/renv/library/R-4.0/x86_64-w64-mingw32\")\n\n\n\n\nUpdating packages in renv\n\nTo update a single package run:\nrenv::update(\"dplyr\")\nTo update all packages run:\nrenv::update()\n\n\n\nInstalling old package versions in renv\n\nThis is surprisingly neat to do. Let’s say you wanted to roll back to version 1.0.2 of dplyr, you would run the following:\nrenv::install(\"dplyr@1.0.2\")"
+ "objectID": "writing-visualising/dashboards.html#peer-review",
+ "href": "writing-visualising/dashboards.html#peer-review",
+ "title": "Public dashboards",
+ "section": "Peer review",
+ "text": "Peer review\nPeer review is a quality assurance activity, where an analyst other than the original author, views and tests the usage of a product or specific piece of code. This allows a fresh pair of eyes to take a look at your work. It validates that you have taken a correct approach and may highlight errors. This constructive feedback helps you to improve the quality. It provides confidence in your work, and ensures that it is fit for purpose.\n\nDashboards must always be peer reviewed within the team they are created.\nDashboards should also be peer reviewed by analysts outside of the subject area of the team.\n\nThe Central Statistics Unit has a number of analysts experience with R Shiny dashboards and are happy to review any dashboards created, contact statistcs.development@education.gov.uk if you’re interested in this. For more guidance on how to peer review, see the peer review section of the Duck Book."
},
{
- "objectID": "learning-development/r.html#quick-reference-lookup",
- "href": "learning-development/r.html#quick-reference-lookup",
- "title": "R",
- "section": "Quick reference lookup",
- "text": "Quick reference lookup\n\nIf you want a useful guide for R syntax or functions, then look no further than the R cheat sheets, these can be an invaluable point of reference. Below we’ve included a few particularly relevant ones:\n\nIntroduction to the RStudio environment\nBase R\nImporting data into R\ndplyr for data manipulation\nstringr for string manipulation\nRegex\nRMarkdown\nRShiny\nggplot2 for data visualisations\npurrr for applying functions"
+ "objectID": "writing-visualising/dashboards.html#internal-only-dashboards",
+ "href": "writing-visualising/dashboards.html#internal-only-dashboards",
+ "title": "Public dashboards",
+ "section": "Internal only dashboards",
+ "text": "Internal only dashboards\nIt’s possible to publish dashboards that are only accessible to those on DfE kit, to do this you will need to publish via the departments RSConnect servers.\nYou will need:\n\nA finished app, in line with the guidance on this page\nThe code to be in a Git repository in the dfe-gov-uk Azure DevOps space\n\nTo publish the app, you’ll need to set up a pipeline in Azure DevOps, guidance for how to do this can be found in the R community teams area.\nAccess to applications on RSConnect are locked down by default, once the pipeline is set up and you’ve deployed the app you’ll need to request for its access to be opened up by using an RSConnect request on ServiceNow.\nIf you’re running the app from an internal database, you’ll need to contact the database owner to set up a local login, and then store those as variables against your specific app in rsconnect. You can raise a request to do this via ServiceNow and selecting ‘Change app variables’."
},
{
- "objectID": "learning-development/r.html#other-resources",
- "href": "learning-development/r.html#other-resources",
- "title": "R",
- "section": "Other resources",
- "text": "Other resources\n\nHere is another free introduction to R course by Quantargo.\nR Markdown: The Definitive Guide, hopefully this one should be relatively self-explanatory!\nData science in education provides a heavily detailed guide for beginners in R learning to process data, with some well written out sections that may be of interest.\nHandy guide to collapsing and sectioning R code for easy navigation in RStudio.\nHere are 5 handy tidyverse functions that you should know if you’re using R to process data. Number two is especially useful for those processing wide data into a tidy format!\nMoJ have produced guidance on writing functions in R\nIf you’re wondering how best to make the jump to R from Excel and SQL, take a look at this coffee and coding presention by David Sands.\nMalcolm Barrett has done some slides on dplyr, ggplot2, and using purrr which may be useful if you’re looking at learning more about any of those packages.\nAlso check out the janitor package, it has some particularly powerful functions that are worth a look for tidying and QA’ing data."
+ "objectID": "writing-visualising/dashboards.html#public-dashboards",
+ "href": "writing-visualising/dashboards.html#public-dashboards",
+ "title": "Public dashboards",
+ "section": "Public dashboards",
+ "text": "Public dashboards\nThe majority of dashboards made to support and augment our Official Statistics will be public facing. For public facing shiny apps you should publish via shinyapps.io. The statistics development team manage a subscription for this and can help you get set up.\nYou will need:\n\nA finished app that meets the accessibility and styling standards (see our Dashboard procedure checklist)\nCode in the dfe-analytical-services GitHub repo\nApproval from your DD\nIf the data underlying the dashboard is currently unpublished, you will need to create dummy data to use in GitHub until the data becomes published (see dummy data guidance section).\n\nTo set up a new app, send the above to statistics.development@education.gov.uk. If your code is not yet hosted in the dfe-analytical-services area you can request for the repository to be moved at the same time as sending approvals.\n\n\nDummy data\n\nWhen creating a public dashboard, all code and data will be hosted in the dfe-analytical-services GitHub repo. This is a public repo, meaning anything stored in here is publicly available. Therefore if you are creating a dashboard with previously unpublished data, you should provide dummy data for the GitHub repo, and only add the real data on your publication date.\nYour dummy data should:\n\nUse the exact same structure and variable names as your real data\nUse random values in place of real values (one example of how to do this is using the rnorm() function)\nSimulate the types of suppression, missing data or anomalies seen in your real data, to ensure the dashboard can account for these.\n\n\n\n\nTesting with unpublished data\n\nWhile you must use dummy data in your GitHub repo, it is understandable that we should test the dashboard works with the real data before it goes live.\nThis can be done using the .gitignore file alongside the datafiles_log.csv and commit hooks explained in the Stopping accidental data uploads guidance. You can view the example files our template repository.\nThe .gitignore file is a plain text file that tells git to ignore specified files in commits and pushes to the repo. Therefore, the first step when wanting to test unpublished data in a dashboard is to add the unpublished data file names to the .gitignore file.\nAdding the file name alone will ensure it is ignored no matter where in the project area you save it (i.e. in any folder). Once this is done, you can add your unpublished data file to your local area, run the app locally, and make edits/commits without uploading the data to Github.\nThis .GitIgnore guidance page has great guidance on how you can utilize wildcards to capture all the files you might want to ignore.\nIf you have any questions on this process please do contact us at statistics.development@education.gov.uk."
},
{
- "objectID": "learning-development/r.html#excel-functions-in-r",
- "href": "learning-development/r.html#excel-functions-in-r",
- "title": "R",
- "section": "Excel functions in R",
- "text": "Excel functions in R\nR can do everything you do in excel, but takes out the human error. The reference table below shows how you would carry out popular Excel commands in R.\nR comes in with a built-in dataset called “iris”. We’ll use this for all examples so you can recreate them in your local area.\nREMEMBER: R is case sensitive, so all references to column names/entries need to be as-is in the dataset you are looking at. Functions exist that can translate all your columns to lower or snake case for ease!\n\n\n\n\n\n\n\n\nCommon Excel Task\nExample with iris \nHow to do in R with dplyr\n\n\n\n\nSelect specific columns\nSelect only species and petal length\niris %>% select(Species, Petal.Length)\n\n\nList unique entries in field (column)\nFind the unique entries for the “Species” column in iris\niris %>% select(Species) %>% distinct()\n\n\nFilter/select based on criteria\nFilter for sepal length >4 and sepal width <2.5, but NOT “versicolor” species\niris %>% filter(Sepal.Length > 4 & Sepal.Width <2.5 & Species != \"versicolor\")\n\n\nFilter for multiple criteria in same column\nFilter for all “setosa” and “versicolor” species\niris %>% filter(Species %in% c(\"setosa\", \"versicolor\")\n\n\nIf else with OR\nCreate new column called “size_group” based on length or width of petal\niris %>% mutate(size_group =if_else( Petal.Length > 4 | Petal.Width >1.5, \"Large\", \"Small\"))\n\n\nMultiple if else\nCreate new column called “flower_price” based on species and petal length\niris %>% mutate(flower_price = case_when( Species == \"setosa\" & Petal.Length > 1.5 ~\"top band\",Species == \"versicolor\" & Petal.Length < 4 ~\"low_band\", TRUE ~ \"mid_band\"))\n\n\nCOUNTIF\nCount the number of species if they have a petal length >1.5\niris %>% filter(Petal.Length > 1.5 ) %>%group_by(Species) %>% count()\n\n\nSUMIF\nSum petal width of species if sepal width <3\niris %>% filter(Sepal.Width <3) %>%group_by(Species) %>%summarise(Petal.Width = sum(Petal.Width))\n\n\nVLOOKUP\nLookup to a table called “lookup”\niris %>% left_join(lookup, by.x=\"Species\", by.y =\"plant_species\")\n\n\nOrder by\nOrder dataset by descending petal width\niris %>% arrange(desc(Petal.Width))\n\n\n\nMore tips for moving from using Excel to using R can be found in the excel-to-R wiki."
+ "objectID": "writing-visualising/dashboards.html#how-to-start",
+ "href": "writing-visualising/dashboards.html#how-to-start",
+ "title": "Public dashboards",
+ "section": "How to start",
+ "text": "How to start\nThere are a lot of resources already available to support you when working with R Shiny, if you’re new, then this blog post provides a gentle introduction.\nOur template shiny app repository is a useful starting point for all public facing dashboards in the department. Please see the dashboard template section for guidance on what the dashboard template includes and how to use it.\nFor more advanced shiny knowledge it’s worth taking a look at the guide to engineering production-grade shiny apps.\nFinally, you should seek to make use of the community that is already out there, see what others are doing, ask them questions and for advice on any decisions or problems that you’re facing, and share what it is that you’re doing. - The Statistics Development Team are experienced with R Shiny and happy to help or offer advice, and there is the DfE R community on teams. - There is a bank of tips and tricks stored in the DfE R teams area too. - Going beyond DfE there’s a wealth of resources and communities online, including Stack Overflow discussions, cross government slack channels (e.g. #R and #Shiny on govdatascience.slack.com) and even tweets about R Shiny on twitter.\n\n\nDashboard procedure checklist\n\nThis checklist outlines the standard procedure for teams who are wishing to produce a public R shiny dashboard.\nGetting set up:\n\nCreate an account on GitHub\nAsk the Statistics Deveopment Team to create you a repository in the DfE analytical services area, providing the name of the dashboard and the GitHub accounts of anyone who will be contributing to the code. You should aim to have two analysts working on the code development and a line manager for review purposes. Further colleagues with review responsibilities (policy colleagues, G6 and above, etc.) can be given access to a demo-site, rather than the repository (see guidance for this below in ‘Setting up a development/demo dashboard area’).\nClone the repo to your device so you can develop your code. Open the repo page in GitHub, click the green ‘Code’ button, and copy the URL it provides. Then, open R Studio on your device, click file > new project > version control > git, paste the repository URL you copied from GitHub, give your local project area a name, and choose where to save it (i.e. on your computer’s C: drive, outside of the OneDrive-synced folders).\n\nOnce you’re set up, there are certain parts of the code you need to update:\n\nIn the global.R script, update all of the site URL’s and EES publication names to your own.\nIn the ui.R script, update the tags$title(), the information in the meta_general(), and the secondary text in shinyGovstyle::header().\nGot to the .github > workflows folder, and open the deploy-shiny.yaml file. At the bottom, update the appName in rsconnect::deployApp() - this is what will appear in the URLs (i.e. must align with the links in global).\nUpdate the ‘Support and feedback’ tab (in R > standard_panels.R) with your teams information. We also recommend creating a feedback form for users of your dashboard, and adding a link to that on this page.\nUpdate the README.md file (you can do this easily directly in GitHub). This is the file that renders below your repo in GitHub for users to see.\nBegin adding your own dashboard content. If you copy and paste any parts of the code (i.e. to create new tabs) you must change all of the IDs so there are no repeated IDs in the UI, otherwise the app will run with no UI elements. You should add UI and unit tests as you develop your code as a form of automated QA (see our guidance on UI tests and guidance on unit tests).\n\nYou must contact the statistics development team for the following:\n\nTo add the shinyapps.io secret and token to your GitHub repo, therefore enabling the app to be hosted on shinyapps.io.\nTo create an area for your team in Google Analytics, to track the user analytics of your dashboard.\n\nSetting up a development/demo dashboard area:\n\nWhile developing your dashboard, you may want a private, demo-version to share with policy or senior colleagues for review and feedback. This version must use either published data or dummy data and can not use unpublished data, since this cannot be uploaded to GitHub until the day of publication (see our dummy data guidance for public dashboards).\nEnsure that prior to contacting the statistics development team, you have updated all of the URL’s and other items listed above.\nYou must contact the Statistics Deveopment Team to add the shinyapps.io secret and token to your GitHub repo, therefore enabling the app to be hosted on shinyapps.io. Once this is done you will have a browser link you can use to access the dashboard. We can make this private such that there is a list of approved viewers who must log in to view the dashboard - please provide the email addresses of any colleagues who you wish to have access to the private version during development.\n\nYou must have done the following before a dashboard can be published (the statistics development team must review and approve that these have been met):\n\nAccessibility testing the dashboard and updating the accessibility statement. The accessibility testing guidance later in this section outlines how teams can do this.\nYou should test how your dashboard appears and performs on mobile devices. you can do this by opening your dashboard in chrome/edge, right clicking anywhere on the page and selecting ‘inspect’. you will then see a new panel on the right hand side of your screen. To see your dashboard how a mobile user would, click the mobile/tablet button (see the image below).\n\n\n\nSetting up UI and unit tests (UI tests are a part of the automated QA process and will run via GitHub actions each time you create a pull request). See our guidance on UI tests and guidance on unit tests.\nPerformance testing your dashboard. See our guidance on performance testing.\nThe underlying data for the dashboard must still be uploaded to EES in the tidy data format to be used in the table tool (check this using our data screener)\nDecide where you are going to provide the public the link to your dashboard. You can easily add a link to your EES publications. If you have a draft release that is going out on the same day as the dashboard, you can add the link into your EES draft while the shinyapps.io page is still private. This is because the link will not change when the dashboard is made public, the log-in screen will simply be removed."
},
{
- "objectID": "learning-development/r.html#sql-functions-in-r",
- "href": "learning-development/r.html#sql-functions-in-r",
- "title": "R",
- "section": "SQL functions in R",
- "text": "SQL functions in R\nR can do a lot of the things that are possible in SQL. The reference table below shows how you would carry out some popular SQL commands in R.\nREMEMBER: R is case sensitive, so all references to column names/entries need to be as-is in the dataset you are looking at. Functions exist that can translate all your columns to lower or snake case for ease!\n\n\n\n\n\n\n\nCommon SQL Task\nHow to do in R (with dplyr)\n\n\n\n\nSELECT * FROM TABLE\ntable %>% select()\n\n\nSELECT ColA, ColB, ColC FROM TABLE\ntable %>% select(ColA, ColB, ColC)\n\n\nSELECT DISTINCT ColA FROM TABLE\ntable %>% select(ColA) %>% distinct()\n\n\nTABLE A LEFT JOIN (TABLE B) ON TABLEA.x = TABLEB.y\ntableA %>% left_join(TableB, by = c(x = y))\n\n\nCASE WHEN x = 1 THEN 1, WHEN x =2 THEN 2, ELSE 0 END AS New_column_name\n%>% mutate (New_column_name = case_when (x == 1 ~ 1, x == 2 ~ 2, TRUE ~ 0))\n\n\nCONCAT(LEA, ESTAB) AS LAESTAB\n%>% mutate(LAESTAB = paste0(LEA, ESTAB))\n\n\nSELECT COUNT(*) FROM TABLE\ntable %>% nrow()\n\n\nSELECT COUNT(ColA) FROM TABLE\ntable %>% count(colA)\n\n\nSELECT Date_column = CONVERT(DATE, Date_column) FROM TABLE\ntable %>% mutate(Date_column = as.Date(Date_column))\n\n\nSELECT Number_column = CONVERT(INT, Number_column ) FROM TABLE\ntable %>% mutate(Number_column = as.numeric(Number_column))\n\n\nSELECT String_column = CONVERT(VARCHAR, String_column ) FROM TABLE\ntable %>% mutate(String_column = as.character(String_column))\n\n\nDROP TableA\nrm(TableA)\n\n\n\nMore tips for moving from using SQL to using R can be found in the SQL-to-R wiki."
+ "objectID": "writing-visualising/dashboards.html#standards-to-follow",
+ "href": "writing-visualising/dashboards.html#standards-to-follow",
+ "title": "Public dashboards",
+ "section": "Standards to follow",
+ "text": "Standards to follow\nWe expect all dashboards to follow a minimum set of standards to ensure coherence between our products and a minimum standard of quality for our end users.\nThese standards are constantly evolving, and all feedback and contributions are welcome, contact us at statistics.development@education.gov.uk.\n\n\nAccessibility testing\n\nIn line with recent legislation for public sector websites, all dashboards need to meet the latest Web Content Accessibility Guidance.\nAs a minimum we expect all dashboards to be checked using the two tools below (lighthouse and shinya11y) and have an accessibility statement before being published.\nYour statement should be written in line with the accessibility statement guidance, and you can make use of a template accessibility statement provided by .gov.uk.\nLighthouse in Google Chrome is an easy way to quickly rate your accessibility. Open your app in Chrome, right click anywhere on the page and select “Inspect”. From there, navigate to “Lighthouse” in the top grey bar, and click “Generate report”. This generates scores for accessibility, best practices and SEO for your application or web page:\n\nTo complement this, we recommend also using the shinya11y package to look at accessibility across your dashboard. Simply install and load the package, then include use_tota11y() at the top of your UI script. This brings up an interface that helps you examine different accessibility aspects in your app, like colour contrast, alt-text and what screen readers will detect:\n\nAutomated tools can’t check for everything and there’s no substitute for giving your dashboard a manual check. You should consider what users need from your dashboard and ensure that that information is accessible to all. For example, can someone using a screen reader get the same information as you can see on the screen from a downloadable csv or from the alt text that you have provided? Everything that you can see, you should also be able to read with a screen reader. Use the Edge narrator tool to test out how your dashboard works in practice.\nOne example of a dashboard for Official statistics that meets these regulations is the SCAP LA school places scorecards app. Their accessibility statement is clearly labelled, explains what checks have been carried out, what the known limitations are and the plans in place to fix them:\n\n\n\n\n\nStyling\n\n\nAll dashboards should have a link to the source code, and information about who to contact for feedback/issues. You should be familiar with and follow the gov.uk style guide as appropriate.\nTeams should pick from the gov.uk design system colours when creating dashboards. The style sheet in our template repository makes use of these colours, though this can be extended if needed to suit your needs. Be careful when choosing colours, and make sure that colours have sufficient contrast to be placed next to each other to meet WCAG AA standards using the colour contrast checker.\nCharts using ggplot2 and plotly can take custom palettes using the scale_fill_manual() function, e.g. below:\n# Create colour palette using recommended colours (based on gov.uk design system)\n\ndfe_colours <- c(\n \"#12436D\", #`blue`\n \"#F46A25\",#`orange`\n \"#801650\",#`maroon`\n \"#28A197\" #`turquoise`\n )\n\n# Create chart using palette\n\nggplot(aes(y=value, x=\"\",\n fill = factor(rating),\n text = paste(rating, \": \", value, \"%\"))) +\n geom_bar(stat=\"identity\", position = position_fill(reverse = TRUE))+\n scale_fill_manual(values = dfe_colours)\n\nFurther examples are given on the Using Explore Education Statistics guidance page.\nYou can use in-line CSS to make these style changes, or make edits to the style sheet. Editing the style sheet is a neater solution if you want to apply changes across your app, and keeps your UI code clean. You can find out how styles are configured by running your app, right clicking and selecting “Inspect” to view the element you wish to change. You can then find these lines in the CSS stylesheet and change a number of things including fonts, font colours and background colours.\nThe SCAP LA school places scorecards app is one example of a public facing dashboard that meets these styling requirements.\n\n\n\n\nTesting R Shiny\n\nTo ensure that they are reliable, dashboards should have an appropriate amount of automated tests before they can be published. We recommend using a mix of UI and unit tests, but the number and type of tests that you need to run will depend on the content of your application.\nFurther guidance on setting up testing can be found in the DfE good code practice guide. Also see our guidance on testing R code on the learning resources page.\n\n\nPerformance testing\n\nPerformance profiling represents a chance to breakdown how different elements within your dashboard perform relative to each other. This consists of running through a set of actions on a local run of your dashboard and timing each one. A useful tool for performing these tests is profvis, which carries out all the timings and visualisation of the results for you, whilst all you have to do is run the app and step through the actions that you want to profile.\nThe basic steps are as follows.\nWith your dashboard repository open in RStudio, run the following commands from the R console\ninstall.packages(\"profvis\")\nprofvis(shiny::runApp())\nThen go through a set of interactions in your dashboard (e.g. navigate to the dashboard page, step through changing the selected choices in each input field, switch between any different tabs and cycle through the relevant inputs again). Once you’ve performed a representative range of actions in the dashboard, close the window and go back to the RStudio window.\nAfter several seconds (depending on the range of actions you performed), a visualisation will appear in the main RStudio editor window showing the results of the profiling.\nThe profvis results are shown in the flame profiling chart. The top half of the chart shows any processing time measurements above 10ms for each line of code. It can be useful to focus in on lines that have measured times above 10ms by selecting “Hide lines of code with zero time” from the options menu in the top right.\n\nLines with timings of greater than about 5-100ms may warrant further investigation to see if they warrant some optimization. Common routes to optimising code are:\n\navoid any data processing in server.R, e.g. avoid using the summarise() function;\nuse reactive() elements to minimise repeated calls of given functions;\nuse memoise() to cache results of more intensive functions (e.g. ggplot) to minimise repeated calls.\n\nThe documentation for the profvis package can be found here: Profvis documentation\nWhilst profvis can help identify any bottlenecks, this will ideally be complemented in the future by full load testing of dashboards, whereby the behaviour of dashboards under real-life high demand scenarios can be tested. However, this type of testing is unavailable whilst our applications are hosted on ShinyApps. We will offer support on load testing once we move dashboards on to our own server systems.\n\n\n\nUI Tests\n\nUser interface (UI) tests should be used for all apps. These tests should check that:\n\nYour app loads up correctly\nContent appears as expected when you switch between tabs\nContent updates as expected when you change inputs\n\nThe shinytest package is a really easy way to create these tests on an RShiny app. Simply load in the package and run shinytest::recordTest() to open up the testing environment to get started.\nEach test should produce one “expected results” JSON file. You can track particular elements of your app by holding CTRL and clicking on an element in the testing window.\nFor example, the SCAP LA school places scorecards app has tests that check that the app functions as expected when different combinations of filters are selected.\nEach test is separated out by comments, clearly stating what is being tested. - app\\(setInputs()** tells the UI test to select an input in the app - **app\\)snapshot() tells the UI test which outputs to capture (i.e. which outputs you’d expect to change and want to test)\n\nThe expected outputs are saved in a UI_tests-expected folder, and every time you run a UI test, the outputs will be compared to these expected snapshots.\nAn example of a basic shiny test to check that your app loads up correctly can be found in our shiny template repo.\n\n\n\nUnit Tests\n\nUnit tests should be used for apps that contain custom functions created from scratch, or that rely on existing functions being combined to interact in a certain way.\nUnit testing checks, for a set of defined variables, that the output is always the same when you run your custom function/s. This ensures that if slight tweaks are made to your function, or functions within packages you are using are updated, that your custom function still works.\nFor example, our QA app has many custom functions that we create to screen files before upload to EES. We have a series of dummy files that we have created to either pass or fail certain tests. Unit tests are written to check that these files always pass or fail as expected.\n\n\n\nTests and deployment\n\nBoth UI and unit tests need to be added to your app’s deployment pipeline. This ensures that the app will not be published if any of the tests fail.\nWe recommend using Github Actions to deploy the latest code whenever a push to your master branch occurs - this ensures that the published version of your app stays up to date.\nYou should also use GitHub Actions to run the automated tests for your app, which we recommend are done on at least every pull request.\nIf you’ve started from our template repository then all of this will be mostly set up for you to tweak and expand on, but if you haven’t then you’ll need to add the yaml files from the .github/workflows folder to your repository.\nGitHub actions are already well documented and their own guidance should be able to walk you through anything you need. That being said, if there’s anything else you’d like to see here let us know.\n\n\n\n\n\nConnecting to databases\n\nWe don’t yet have a database server set up that can be accessed by public facing dashboards, though we are working to put this in place. In the meantime, there a few alternative options for storing data that the dashboard can run off.\n\nStore the data within the repo (e.g. CSV files in a /data folder). Note that only published data should be stored in a repo. If your working on a dashboard that uses unpublished data then please see the dummy data section for guidance.\nUse Google sheets\nUse Dropbox\n\nIf you are running an internal-only app then you can connect to our internal SQL servers, information on how to do this is in the R community teams area.\n\n\n\nSecure variables\n\nSee our Git page for guidance on storing secure variables in repositories."
},
{
- "objectID": "learning-development/r.html#tips-for-using-r",
- "href": "learning-development/r.html#tips-for-using-r",
- "title": "R",
- "section": "Tips for using R",
- "text": "Tips for using R\nA selection of handy bits of code and workarounds for common issues. More useful code snippets can also be found in our github repo\n\n\nSpecifying a version of R to use\n\nThis can be done most easily by navigating in RStudio through Tools > Global options > General > Basic > R version (change). It’s likely you’ll need to restart RStudio for the changes to take affect.\n\n\n\nRounding\n\nThe base R function of round() rounds 5’s downwards. To round them upwards you can create a custom function like the one below:\nroundFiveUp <- function(x, n){ \n z = abs(x)*10^n \n z = z + 0.5 + sqrt(.Machine$double.eps) \n z = trunc(z) \n z = z/10^n \n positiveNegative = sign(x) \n return(z * positiveNegative) \n}\nThis function can also be found in the dfeR R package.\n\n\n\nPassing variables as arguments\n\nThis can be worked around by using a combination of eval() and parse(), as shown in the below function:\nshowFilterLevels <- function(data, meta) {\n filters <- meta %>%\n filter(col_type == \"Filter\") %>%\n pull(col_name)\n\n levelsTable <- function(filter) {\n return(eval(parse(text = paste0(\"data %>% select(\", filter, \") %>% distinct()\"))))\n }\n\n output <- lapply(filters, levelsTable)\n\n return(output)\n}\n\n\n\nReverse additive filters\n\nYou might want to filter your dataset based on multiple negative conditions. Normally to filter on multiple conditions, you would use filter(condition1 & condition2). The “filter” function does not work well with negative conditions (i.e. filtering for cases where condition 1 and condition 2 are not met). Instead, you can use subset(!(condition1 & condition2).\n\n\n\nFile locations\n\nStruggling to get files to talk to one another, or get code to find and use another R script? Use here::here() and marvel at it’s wondrous ability to magic away issues.\n\n\n\nInterweaving vectors\n\nThere’s an easy way to interweave multiple vectors into one single vector using c(rbind()). The example below shows two vectors, but you can have even more if you need.\n#Two vectors, x and y\nx <- 1:3\ny <- 4:6\n\n#Run code to interweave\nc(rbind(x, y))\n\n#Output below\n# [1] 1 4 2 5 3 6\n\n\n\nMaking charts interactive\n\nWhen pulling ggplot charts into RMarkdown reports, you can consider making them even more user-friendly and interactive with plotly. Further information on how to make your charts interactive with plotly can be found online.\n#Simple ggplot chart called \"p\"\np <- ggplot(dat, aes(x=xvar, y=yvar)) +\n geom_point(shape=1) # Use hollow circles\n\n#Apply ggplotly() to it to make it interactive\nfig <- ggplotly(p)\n\n\n\n\nReplace all values with another\n\nHave you ever needed to replace every value in your data with another? This can come in handy when you are looking at suppression, e.g. converting all NAs to “z” or all values under a certain threshold to “c”.\ndata %>% mutate_all(~ replace(., . == \"Value to replace\", \"Replacement\"))\n\n\n\n\nTemporary groups\n\nThe group_by() function in dplyr is really useful, but can be fiddly if you only want to use it for one operation in a chunk of code. The with_groups() function from dplyr lets you do this, saving you having to group and ungroup data each time.\ndata %>% mutate(annual_average = with_groups(time_period, mean))\n\n\n\n\nFinding package dependencies\n\nOften we’ll take chunks of code and reuse them for new projects. This can lead to building up a long list of packages to install, not all of which end up being used in your new code. The NCmisc package is a really handy way to check which packages and functions are used in your code.\nFirstly, load up all the packages the code has library() commands for, then run the following:\nlist.functions.in.file('your-filename-here.R', alphabetic = TRUE)\n\n\n\nVisualise dependencies\n\nThe depgraph package allows you to plot a graph of all the dependencies in your R project, which can be a useful tool to help you cut down on the number of package dependencies. Briefly, in these graphs you can look for “hot spots” in the network (big bright dots), which represent packages that have many upstream dependencies but are potentially easy to remove because they have few downstream dependencies (that is, only your package depends on them).\nplot_dependency_graph(\n pkg = multibridge_pkg\n , suggests = FALSE\n , option = \"cividis\"\n)\n\n\n\n\nReproducible random numbers\n\nThe set.seed() function generates a sequence of random numbers, starting from the value you define in the brackets. This ensures you get the same sequence of random numbers each time you run set.seed() with the same value, which is helpful to test that your results are reproducible.\n# random sampling\n> sample(LETTERS, 5)\n[1] \"K\" \"N\" \"R\" \"Z\" \"G\"\n> sample(LETTERS, 5)\n[1] \"L\" \"P\" \"J\" \"E\" \"D\"\n\n# reproducible random sampling\n> set.seed(42); sample(LETTERS, 5)\n[1] \"Q\" \"E\" \"A\" \"J\" \"D\"\n> set.seed(42); sample(LETTERS, 5)\n[1] \"Q\" \"E\" \"A\" \"J\" \"D\"\n\n\n\n\nAutomatic logging\n\nThe tidylog package is a really useful tool for providing automated feedback on dplyr and tidyr operations.\nlibrary(tidylog)\n\nfiltered <- filter(mtcars, cyl == 4)\n#> filter: removed 21 rows (66%), 11 rows remaining\nmutated <- mutate(mtcars, new_var = wt ** 2)\n#> mutate: new variable 'new_var' (double) with 29 unique values and 0% NA\n\n\n\nRunning SQL scripts from R\n\nR can be used to execute SQL scripts to extract data from a database as well as querying the database directly via R. For using R to execute a SQL script you’ll need the SQL script/s to be in your R Project and to make a connection via R to the database.\n# Library calls ====\n\nlibrary(odbc)\nlibrary(DBI)\n\n# DB connection ====\n\ncon <- DBI::dbConnect(odbc::odbc(),\n Driver = \"ODBC Driver 17 for SQL Server\",\n Server = \"server_name\",\n Database = \"database_name\",\n UID = \"\",\n PWD = \"\",\n Trusted_Connection = \"Yes\"\n)\n\n# Function to read in sql scripts ====\n\ngetSQL <- function(filepath){\n con = file(filepath, \"r\")\n sql.string <- \"\"\n \n while (TRUE){\n line <- readLines(con, n = 1)\n \n if ( length(line) == 0 ){\n break\n }\n \n line <- gsub(\"\\\\t\", \" \", line)\n \n if(grepl(\"--\",line) == TRUE){\n line <- paste(sub(\"--\",\"/*\",line),\"*/\")\n }\n \n sql.string <- paste(sql.string, line)\n }\n \n close(con)\n return(sql.string)\n}\n\n# Execute SQL query and pull into R ====\n\nmy_table <- dbGetQuery(con, getSQL(\"my_query_script.sql\"))\n\nIf you’re struggling to get your SQL scripts to execute from R, try adding the following two lines to the top of your SQL script to force the formatting required for it to work:\nSET ANSI_PADDING OFF\nSET NOCOUNT ON;\n\n\n\nCan’t find make error\n\nThis error will appear when trying to restore old package versions using renv::restore(). It is usually due to Rtools not being properly installed, and therefore your system is struggling to install the older version of the package that hasn’t been provided by CRAN.ware centre.\nThe solution takes a few minutes, but is relatively straightforward, you can install Rtools as a direct download from CRAN. On there, pick the right version of RTools for your version of R, download the file and install.\nSet the install location to your c drive, for example, if installing for R4.2.2, install RTools42 to C:\\rtools42 (this is the default location).\nOnce it’s installed close and reopen RStudio and run renv::restore() again - it should hopefully now restore and install your packages successfully!\n\n\n\nPackages not downloading due to “wininet” error (renv 0.17)\n\nPackages are generally downloaded from CRAN using one of two methods: wininet or curl. Since renv version 0.17, analysts potentially hit an error associated with renv defaulting to wininet, whilst R defaults to curl, causing package downloads to fail. To fix this issue, analysts should change their default download method with renv to be curl. This can be done across all R-projects on your machine, by setting the default in your global R environment file. To do this, open up a bash terminal and enter the following command:\necho 'RENV_DOWNLOAD_METHOD = \"curl\")' >> ~/.Renviron\nThen restart R-Studio and downloads with renv should now succeed."
+ "objectID": "writing-visualising/dashboards.html#publishing-your-dashboard",
+ "href": "writing-visualising/dashboards.html#publishing-your-dashboard",
+ "title": "Public dashboards",
+ "section": "Publishing your dashboard",
+ "text": "Publishing your dashboard\n\nDfE Shiny applications are published via the DfE Analytical Services shinyapps.io account. You need to alert the statistics development team of any new dashboard publication as early in development as possible and keep us updated on expected publication date. Update the stats development team on any subsequent data or major functional updates to the dashboard publication at least a week prior to re-publishing with the update. Deploying to shinyapps requires the DfE platform codes to be entered into the repository secrets area of your app. This needs to be done by the stats development team. Authorisation of a publication should be requested from the relevant G6 or DD and the stats development team (with the former authorisation e-mail being forwarded on to the Statistics Deveopment Team).\nIf you are publishing a dashboard using already published data, then all of your code and data should be on GitHub. You may have decided to password-protect the dashboard URL, in which case, you should make the Statistics Deveopment Team aware of your publication date so that they can remove the password-protection at 9:30 on publication day, making the dashboard visible to the public.\nIf you are publishing a new dashboard for the first time that uses unpublished data, then you should have followed the guidance on using dummy data. This means that the unpublished data should not be added to GitHub until the day of publication. You should follow steps 1-9 in the below section on the day before and day of publication.\n\nBe sure to read the guidance carefully, do not commit or push unpublished data to a GitHub repo before the day of the publication of the data. If you think you may have done this by accident, contact Statistics Deveopment Team immediately with the full details of what has been uploaded to GitHub and when.\n\n\n\nUpdating data in a dashboard\n\nAs mentioned in the public dashboards section, a public dashboard should not be updated with unpublished data until the data is published. However, it is possible to clone the repo and run it locally with unpublished data for testing purposes (see our guidance on testing with unpublished data). This guidance applies to both adding real data to a dashboard that previously used dummy data, and to adding updated data to an existing dashboard.\nOn the day prior to publication:\n\nCreate a local branch which you will use to update the data. To do this open your local version of the repo, navigate to the Git window, ensure the current branch is main (see 1) as this is the code producing the current live version of the app. If you’re happy the current branch is main, then do a pull to ensure it is up-to-date (see 2). Then click the new branch button (see 3).\n\n\n\nGive your branch a descriptive name, for example, for a data update for publication on a given date, incorporate the date into the branch name (i.e. data_update_01Jan_2023). Click create.\nYou can now add your unpublished data files to the data folder in your local copy of the repository (in the file explorer). If you have already added the data using .gitignore as described in the guidance on testing with unpublished data, you can now remove the file name from the .gitignore file. You can also remove any dummy data files or old data files from the folder.\nNow open the script that reads in data for your dashboard and edit the file paths to point at the new data files.\nrun tidy_code() and run_tests_locally() (see guidance on UI tests). If changes are found when running the tests locally, make sure you look through these differences and understand them. If you are happy with the changes found in the tests (i.e. they are expected due to your updates), update the .json files. These are tested via GitHub actions every time you do a pull request, so you should always run them in advance of any pull request.\nYou can run your app to test that it is working, but do not commit or push this branch yet.\n\nOn the day of publication:\n\nAt 9:30 on publication day, commit and push the changes in your new branch to GitHub. This makes the data files publicly available via GitHub and so should not be done until 9:30 on the day of publication, however this step does not update the dashboard.\nOn the GitHub repo, you should now see a prompt to open a pull request. You should do this, following the template, ticking the boxes to show you have completed tests locally as required and provide details of the changes (i.e. that you’ve updated to include the most recent data, including dates).\nAs you have already completed tests locally (step 5), you do not have to wait for the tests to complete when you open the pull request, you can click to merge into main straight away. This should start the process of updating the data on the dashboard. You can view the progress and time taken to do this by opening the GitHub actions tab and looking at the shiny-deploy action. Once the shiny-deploy action is complete, the dashboard will have updated."
+ },
+ {
+ "objectID": "writing-visualising/dashboards.html#whats-in-the-template",
+ "href": "writing-visualising/dashboards.html#whats-in-the-template",
+ "title": "Public dashboards",
+ "section": "What’s in the template",
+ "text": "What’s in the template\nThe template provides code for a basic interactive dashboard, with an example line chart using drop-downs for education level and geographic levels. It also includes an example accessibility statement (this is a requirement for all public dashboards and must be filled in, see the Accessibility and Accessibility testing sections earlier in this guidance), and a ‘Support and feedback’ page with suggested contact information and links.\nView the latest version of the template dashboard.\n\n\nGoogle analytics\n\nThe template has a google_analytics.html file which is set up to track all basic metrics, plus the inputs from both drop-downs. To set this up with your own app you will need a new tracking tag (the long number in the gtag()) which the statistics development team can provide you with. Please contact us in order to set this up.\nIf you are hosting your dashboard at multiple URLs (i.e. to cope with expected high traffic) then all URLs will be tracked using one tracking tag and analytics for all URLs will all appear as one in the same report.\n\n\n\nData files log\n\nThe datafiles_log.csv file is a record of all of the data files saved in the repo, and whether or not the data is published. You should log all data files here and accurately record whether the data is published, unpublished or reference data.\n\n\n\nStopping accidental data uploads\n\nIf you see any of the following errors, it is because you are trying to commit a data file to the GitHub repo that hasn’t been recorded as published:\nError: data/test_data.csv is not recorded in datafiles_log.csv.\nError: data/test_data.csv is not logged as published or reference data in datafiles_log.csv and is not found in .gitignore.\nThe template uses commit hooks and a datafiles_log.csv file. Commit hooks run automatically whenever you try to commit changes, and prevent any unpublished data being uploaded to the GitHub repo (see the public dashboards and dummy data sections) by checking the datafiles_log.csv file.You should therefore record all data in the datafiles_log and record whether it is published, unpublished or reference dara. If you try to commit a file that is not recorded or is recorded as unpublished, git will not allow the commit. If you would like to save an unpublished data file locally to test the dashboard you should use the .gitignore file to ensure git ignores this file in your commits (see the testing with unpublished data guidance)."
},
{
"objectID": "RAP/rap-expectations.html",
@@ -868,430 +903,395 @@
"text": "Analysts\nAnalysts working on analysis in government will:\n\nuse open-source tools wherever whenever appropriate\nopen source their code\nwork with data engineers and architects to make sure that source data are versioned and stored so that analysis can be reproduced\nlearn the skills they need to implement RAP principles\nengage with users of their analysis to demonstrate the value of RAP principles and build motivation for development\ndeliver their analysis using RAP"
},
{
- "objectID": "RAP/rap-statistics.html",
- "href": "RAP/rap-statistics.html",
- "title": "RAP for Statistics",
+ "objectID": "RAP/rap-support.html",
+ "href": "RAP/rap-support.html",
+ "title": "RAP support",
"section": "",
- "text": "Guidance for how to implement the principles of Reproducible Analytical Pipelines (RAP) into statistics production processes"
- },
- {
- "objectID": "RAP/rap-statistics.html#our-scope",
- "href": "RAP/rap-statistics.html#our-scope",
- "title": "RAP for Statistics",
- "section": "Our scope",
- "text": "Our scope\nWe want to focus on the parts of the production process that we have ownership and control over – so we are focussing on the process from data sources to publishable data files. This is the part of the process where RAP can currently add the most value - automating the production and quality assurance of our outputs currently takes up huge amount of analytical resource, which could be better spent providing insight and other value adding activity. \n\n\n\nIn Official Statistics production we are using RAP as a framework for best practice when producing our published data files, as these are the foundations of our publications moving forward. Following this framework will help us to improve and standardise our current production processes and provide a clear ‘pipeline’ for analysts to follow. This will have the added benefit of setting a clear and defined world of tools and skills required, making learning and development that much clearer and easier. To get started with RAP, we first need to be able to understand what it actually means in practice, and be able to assess our own work against the principles of RAP.\nImplementing RAP for us will involve combining the use of SQL, R, and clear, consistent version control to increase efficiency and accuracy in our work. For more information on what these tools are, why we are using them, and resources to help up-skill in those areas, see our learning resources page.\nThe collection of, and routine checking of data as it is coming into the department is also an area that RAP can be applied to. We have kept this out of scope at the moment as the levels of control in this area vary wildly from team to team. If you would like advice and help to automate any particular processes, feel free to contact us."
- },
- {
- "objectID": "RAP/rap-statistics.html#how-to-assess-your-publication",
- "href": "RAP/rap-statistics.html#how-to-assess-your-publication",
- "title": "RAP for Statistics",
- "section": "How to assess your publication",
- "text": "How to assess your publication\nThe checklist provided in the publication self-assessment tool is designed to make reviewing our processes against our RAP levels easier, giving a straightforward list of questions to check your work against. This will flag potential areas of improvement, and you can then use the links to go to the specific section with more detail and guidance on how to develop your current processes in line with best practice.\nSome teams will already be looking at best practice, while others will still have work to do to achieve the department’s baseline of good and great practice. We know that all teams are starting this from different points, and are here to support all teams from their respective starting positions."
- },
- {
- "objectID": "RAP/rap-statistics.html#where-we-need-to-focus",
- "href": "RAP/rap-statistics.html#where-we-need-to-focus",
- "title": "RAP for Statistics",
- "section": "Where we need to focus",
- "text": "Where we need to focus\nMost teams have already made progress with their production of tidy data files, and the release of the automated screener has now tied up that end point of the pipeline that we are all currently working towards. The standard pipeline for all teams will roughly resemble this:\n\n\n\nThe key now is for us to build on the work so far and focus on how we improve the quality and efficiency of our production processes up to that point. To do this, we need to make a concerted effort to standardise how we store and access our data, before then automating what we can to reduce the burden of getting the numbers ready and see the benefits of RAP. The exact meaning of this will vary within teams."
- },
- {
- "objectID": "RAP/rap-statistics.html#how-to-get-started",
- "href": "RAP/rap-statistics.html#how-to-get-started",
- "title": "RAP for Statistics",
- "section": "How to get started",
- "text": "How to get started\nMeasure your publication against the RAP levels using our self assessment tool. This will give you a good starting point and initial points to work on to progress to the next level of RAP.\nOnce you’ve assessed your publication, have a look through our guidance below to narrow down how you can get started with improving those parts of your process.\nThe Statistics Development Team invites teams to take part in our partnership programme to develop their skills and implement RAP principles to a relevant project. Visit our page on getting started with the partnership programme for more details."
- },
- {
- "objectID": "RAP/rap-statistics.html#all-source-data-stored-in-a-database",
- "href": "RAP/rap-statistics.html#all-source-data-stored-in-a-database",
- "title": "RAP for Statistics",
- "section": "All source data stored in a database",
- "text": "All source data stored in a database\n\n\n\nWhat does this mean?\nWhen we refer to ‘source data’, we take this to mean the data you use at the start of the process to create the underlying data files. Any cleaning at the end of a collection will happen before this.\nIn order for us to be able to have an end-to-end data pipeline where we can replicate our analysis across the department, we should store all of the raw data needed to create aggregate statistics in a managed Microsoft SQL Server. This includes any lookup tables and all administrative data from collections prior to any manual processing. This allows us to then match and join the data together in an end-to-end process using SQL queries.\nAs far as meeting the requirement to have all source data in a database, databases other than SQL may be acceptable, though we can’t support them in the same way.\nWhy do it?\nThe principle is that this source data will remain stable and is the point you can go back to and re-run the processes from if necessary. If for any reason the source data needs to change, your processes will be set up in a way that you can easily re-run them to get updated outputs based on the amended source data with minimal effort.\nSQL is a fantastic language for large scale data joining and manipulation; it allows us to replicate end-to-end from raw data to final aggregate statistics output. Having all the data in one place and processing it in one place makes our lives easier, and also helps us when auditing our work and ensuring reproducibility of results.\nHow to get started\n\nFor a collection of relevant resources to use when learning SQL, see our learning resources page, and for guidance on best practice when writing SQL queries, see the writing code and documentation sections on this page, as well as the guides immediately below on how to setup and use a SQL database.\n\n\nHow to set up a SQL working area\n\nThere are a few different options, depending on where you want your new area to exist. Visit our SQL learning page for details.\n\n\n\nMoving data to different areas\n\nIf your data is already in SQL, you can use this snippet of R code to move tables from one area (e.g. the iStore) to another (e.g. your team’s modelling area) to ensure all data are stored in a database.\nlibrary(odbc)\nlibrary(dplyr)\nlibrary(dbplyr)\nlibrary(DBI)\n\n# Step 1.1.: Connect to source server -------------------------------------------\ncon_source <- dbConnect(odbc(),\n Driver = \"SQL Server Native Client 11.0\",\n Server = \"Name_of_source_server\",\n Database = \"Source_database\",\n Trusted_Connection = \"yes\"\n)\n\n# Step 1.2.: Connect to target server\ncon_target <- dbConnect(odbc(),\n Driver = \"SQL Server Native Client 11.0\",\n Server = \"Name_of_target_server\",\n Database = \"Your_target_database\",\n Trusted_Connection = \"yes\"\n)\n\n# Step 2.1.: Pull the table from the source database\ntable_for_transfer <- tbl(con_source,in_schema(\"schema_name\", \"table_name\")) %>% collect()\n\n# Step 2.2.: Copy table into target database \ndbWriteTable(con_target,\"whatever_you_want_to_call_new_table\", table_for_transfer)\n\n\n\n\nImporting data to SQL Server\n\nThere’s lots of guidance online of how to import flat files from shared areas into Microsoft SQL server on the internet, including this guide.\nRemember that it is important to import them with consistent, thought-through naming conventions. You will thank yourself later.\n\n\n\nHow to grant access to your area\n\nMuch like setting up a SQL area, there are different ways to do this depending on the server your database is in. Visit our SQL learning page for details."
- },
- {
- "objectID": "RAP/rap-statistics.html#standardised-reference-data",
- "href": "RAP/rap-statistics.html#standardised-reference-data",
- "title": "RAP for Statistics",
- "section": "Standardised reference data",
- "text": "Standardised reference data\nThere will of course be cases where some of the data you use is reference data not owned by the DfE, or is available online for you to download rather than in an internal server. There are ways of incorporating this into a reproducible analytical pipeline nicely, sometimes you can even use links/URLs in your code that will always pull the latest data such that you will never need to change the link to reflect updates to the data!\n\n\nRaw files on GitHub\n\nIf you want to use reference data from GitHub, you can use the URL to the raw file. A common example of when you might want to use this would be to use geographic lookup tables that contain names and codes of different geographic levels for EES files (which are available in our data screener repository).\nIf you find the data you’re interested in within a repository, rather than copying, cloning or downloading the data, you should click the ‘raw’ button (see the below screenshot).\n\n\n\nThis should take you to a new window which contains the raw data from the CSV file. Copy the URL from this page, as this is what we can use to pull the latest data into our code.\nYou can now use this URL as you would use a file path in a read.csv() query. For example;\ndevolved_area_lookup <- read.csv(\"https://raw.githubusercontent.com/dfe-analytical-services/dfe-published-data-qa/master/data/english_devolved_areas.csv\")\nThe above code snippet will load in the data as it would with any other CSV, however the benefit is if that data file is updated on GitHub, when you run the code it will always pull the latest version. This is especially useful for look up tables and reference data, and is best RAP practice as it removed the need for any manual steps (like downloading, copying & pasting or manually updating the data/code)!"
+ "text": "This page outlines the support available for RAP in DfE"
},
{
- "objectID": "RAP/rap-statistics.html#processing-is-done-with-code",
- "href": "RAP/rap-statistics.html#processing-is-done-with-code",
- "title": "RAP for Statistics",
- "section": "Processing is done with code",
- "text": "Processing is done with code\n\n\n\nWhat does this mean?\nAll extraction, and processing of data should be done using code, avoiding any manual steps and moving away from a reliance on Excel, SPSS, and other manual processing. In order to carry out our jobs to the best of our ability it is imperative that we use the appropriate tools for the work that we do.\nEven steps such as copy and pasting data, or pointing and clicking, are fraught with danger, and these risks should be minimised by using code to document and execute these processes instead.\nWhy do it?\nUsing code brings numerous benefits, computers are far quicker, more accurate, and far more reliable than humans in many of the tasks that we do. Writing out these instructions saves us significant amounts of time, particularly when it can be reused in future years, or even next week when one specific number in the source file suddenly changes, and also provides us with editable documentation for our production processes, saving the need for writing down information in extra documents.\nReliability is a huge benefit of the automation that RAP brings - when one of the lines of data has to be amended a week before publication, it’s a life saver to know that you can re-run your process in minutes, and reassuring to know that it will give you the result you want. You can run the same code 100 times, and be confident that it will follow the same steps in the same order every single time.\nHow to get started\nSee our learning resources for a wealth of resources on SQL and R to learn the skills required to translate your process into code.\nThere are also two sections below with examples of tidying data in SQL and R to get you started.\nEnsure that any last-minute fixes to the process are written in the code and not done with manual changes.\n\n\nProducing tidy underlying data in SQL\n\nTo get started, here is a SQL query that you can run on your own machine and walks you through the basics of tidying a simple example dataset in SQL.\n\n\n\nTidying and processing data in R\n\nHere is a video of Hadley Wickham talking about how to tidy your data to these principles in R. This covers useful functions and how to complete common data tidying tasks in R. Also worth taking a look at applied data tidying in R, by RStudio.\nUsing the %>% pipe in R can be incredibly powerful, and make your code much easier to follow, as well as more efficient. If you aren’t yet familiar with this, have a look at this article that provides a useful beginners guide to piping and the kinds of functions you can use it for. The possibilities stretch about as far as your imagination, and if you have a function or task you want to do within a pipe, googling ‘how do I do X in dplyr r’ will usually start to point you in the right direction, alternatively you can contact us, and we’ll be happy to help you figure out how to do what you need.\nA quick example of how powerful this is is below, where my_data is processed to create new columns, have column names renamed, have the column names tidied using the janitor package, blank rows and columns removed, data filtered to only include specific geographic levels, and rows rearranged in order, all in a few lines of easy to follow code:\n\nprocessed_regional_data <- my_data %>% \n mutate(newPercentageColumn = (numberColumn / totalPopulationColumn) * 100) %>% \n rename(newPercentageColumn = percentageRate,\n numberColumn = number,\n totalPopulationColumn = population) %>% \n clean_names() %>% \n remove_empty() %>% \n filter(geographic_level == \"Regional\") %>% \n arrange(time_period, region_name)\n\nHelpful new functions in the tidyverse packages can help you to easily transform data from wide to long format (see tip 2 in the linked article for this, as it is often required for tidy data), as well as providing you with tools to allow you quickly and efficiently change the structure of your variables.\nFor further resources on learning R so that you’re able to apply it to your everyday work, have a look at the learning resources page."
+ "objectID": "RAP/rap-support.html#email-support",
+ "href": "RAP/rap-support.html#email-support",
+ "title": "RAP support",
+ "section": "Email support",
+ "text": "Email support\n\nstatistics.development@education.gov.uk\n\n9-5, aim to reply between 1-2 days."
},
{
- "objectID": "RAP/rap-statistics.html#appropriate-tools",
- "href": "RAP/rap-statistics.html#appropriate-tools",
- "title": "RAP for Statistics",
- "section": "Appropriate tools",
- "text": "Appropriate tools\n\n\n\nWhat does this mean?\nUsing the recommended tools on our learning page (SQL, R and Git), or other suitable alternatives that allow you to meet the core principles. Ideally any tools used would be open source, Python is a good example of a tool that would also be well suited, though is less widely used in DfE and has a steeper learning curve than R.\nOpen-source refers to something people can modify and share because its design is publicly accessible. For more information, take a look at this explanation of open-source, as well as this guide to working in an open-source way. In practical terms, this means moving away from the likes of SPSS, SASS and Excel VBA, and utilising the likes of R or Python, version controlled with git, and hosted in a publicly accessible repository.\nWhy do it?\nThere are many reasons why we have recommended the tools that we have, the recommended tools are:\n\nalready in use at the department and easy for us to access\neasy and free to learn\ndesigned for the work that we do\nused widely across data science in both the public and private sector\nallow us to meet best practice when applying RAP to our processes\n\nHow to get started\nGo to our learning page to read more about the recommended tools for the jobs we do, as well as looking at the resources available there for how to build capability in them. Always feel free to contact us if you have any specific questions or would like help in understanding how to use those tools in your work.\nBy following our guidance in saving versions of code in an Azure DevOps, we will then be able to mirror those repositories in a publicly available GitHub area."
+ "objectID": "RAP/rap-support.html#technical-workshops",
+ "href": "RAP/rap-support.html#technical-workshops",
+ "title": "RAP support",
+ "section": "Technical workshops",
+ "text": "Technical workshops\n\nIn person workshops covering specific technical skills in practice\n3 hours long, with people working in small groups\nRecently circled the sites running an introduction to Git, and introduction to R and RAP.\nContact statistics.development@education.gov.uk to register interest in workshops happening at your site or to request new topics!\n\nWe’re also considering a dedicated G6 / G7 programme to build confidence and set expectations. This may include:\n\nA 2-hour workshop explaining what teams are expected to do when creating and publishing statistics. Including tools and skills required, RAP, using EES, content design, release clearance, UE + ongoing publishing and where teams can go for help.\nThere will be take away materials that cover lines to take and where to get support.\nAll G6s and G7s are expected to do this, with repeats running to ensure everyone gets a chance to attend.\n\nKeep an eye out on teams for this programme being advertised and contact statistics.development@education.gov.uk."
},
{
- "objectID": "RAP/rap-statistics.html#using-run-scripts",
- "href": "RAP/rap-statistics.html#using-run-scripts",
- "title": "RAP for Statistics",
- "section": "Using ‘run’ scripts",
- "text": "Using ‘run’ scripts\nUtilising a single ‘run’ script to execute processes written in other scripts brings a number of benefits. It isn’t just about removing the need to manually trigger different code scripts to get the outputs, but it means the entire process, from start to finish, is fully documented in one place. This has a huge number of benefits, particularly for enabling new team members to pick up existing work quickly, without wasting time struggling to understand what has been done in the past.\n\n\nConnecting R to SQL\n\nIn order to create a single script to run all processes from, it is likely that you will need to use R to run SQL queries. If you are unsure of how to do this, take a look at the materials from Cathy Atkinson’s coffee and coding session on connecting R to SQL using DBI and odbc.\nChris Mason-Thom did another coffee and coding session on this, which you can watch below:\n\n\n\n\n\n\n\nDataset production scripts\n\n\n\n\nWhat does this mean?\nEach dataset can be created by running a single script, which may ‘source’ multiple scripts within it. This does not mean that all of the code to create a file must be written in a single script, but instead that there is a single ‘create file’ or ‘run’ script that sources every step in the correct order such that every step from beginning to end will be executed if you run that single ‘run’ script.\nThis ‘run’ script should take the source data right through to final output at the push of a button, including any manipulation, aggregation, suppression etc.\nWhy do it?\nHaving a script that documents the whole process for this saves time when needing to rerun processes, and provides a clear documentation of how a file is produced.\nHow to get started\nReview your current process - how many file scripts does it take to get from source data to final output, why are they separated, and what order should they be run in? Do you still have manual steps that could introduce human error (for example, manually moving column orders around in excel)?\nYou should automate any manual steps such as the example above. If it makes sense to, you could combine certain scripts to reduce the number. You can then write code in R to execute your scripts in order, so you are still only running one script to get the final output.\n\n\n\nWhole publication production scripts\n\n\n\n\nWhat does this mean?\nThe ultimate aim is to utilise a single script to document and run off everything for a publication, the data files, any QA, any summary reports. This script should allow you to run individual outputs by themselves as well, so make sure that each data file can be run in isolation by running single lines of this script. All quality assurance for a file is also included in the single script that can be used to create a file from source data (see the dataset production scripts section)\nWhy do it?\nThis carries all of the same benefits as having a single ‘run’ script for a file, but at a wider publication level, effectively documenting the entire publication process in one place. This makes it easier for new analysts to pick up the process, as well as making it quicker and easier to rerun as all reports relating to that file are immediately available if you ever make changes file.\nHow to get started\nThe Education, Health and Care Plans production cycle is a good example of a single publication ‘run’ script. They have kept their actual data processing in SQL, but all the running and manipulation of the data happens in R.\nThe cycle originally consisted of multiple SQL scripts, manual QA and generation of final files.\n\n\n\nThe team now have their end-to-end process fully documented, which can be run off of one single R script. The ‘run’ script points at the SQL scripts to run them all in one go, and also creates a QA report and corresponding metadata files that pass the data screener. Each data file can still be run in isolation from this script."
+ "objectID": "learning-development/git.html",
+ "href": "learning-development/git.html",
+ "title": "Git",
+ "section": "",
+ "text": "Guidance and tips for version control with Git"
},
{
- "objectID": "RAP/rap-statistics.html#recyclable-code-for-future-use",
- "href": "RAP/rap-statistics.html#recyclable-code-for-future-use",
- "title": "RAP for Statistics",
- "section": "Recyclable code for future use",
- "text": "Recyclable code for future use\n\n\n\nWhat does this mean?\nWe’d expect that any recyclable code would take less than 30 minutes of editing before being able to run again in a future iteration of the publication.\nWhy do it?\nOne huge benefit that comes with using code in our processes, is that we can pick them up in future years and reuse with minimum effort, saving us huge amounts of resource. To be able to do this, we need to be conscious of how we write our code, and write it in a way that makes it easy to use in future releases for the publication.\nHow to get started\nReview your code and consider the following:\n\nWhat steps might need re-editing or could become irrelevant?\nCan you move all variables that require manual input (e.g. table names, years) to be assigned at the top of the code, so it’s easy to edit in one place with each iteration?\nAre there any fixed variables that are prone to changing such as geographic boundaries, that you could start preparing for changes now by making it easy to adapt in future?\n\nFor example, if you refer to the year of publication in your code a lot, consider replacing every instance with a named variable, which you only need to change once at the start of your code. In the example below, the year is set at the top of the code, and is used to define “prev_year”, both of which are used further down the code to filter the data based on year.\n\nthis_year <- 2020\nprev_year <- this_year - 1\n\ndata_filtered <- data %>% \n filter(year == this_year)\n\ndata_filtered_last_year <- data %>% \n filter(year == prev_year)"
+ "objectID": "learning-development/git.html#what-is-git",
+ "href": "learning-development/git.html#what-is-git",
+ "title": "Git",
+ "section": "What is Git",
+ "text": "What is Git\nIt is a version control software. It is by far the best of its kind and is widely used by software developers and data scientists."
},
{
- "objectID": "RAP/rap-statistics.html#standards-for-coding",
- "href": "RAP/rap-statistics.html#standards-for-coding",
- "title": "RAP for Statistics",
- "section": "Standards for coding",
- "text": "Standards for coding\nCode can be written in many different ways, and in languages such as R, there are often many different functions and routes that you can take to get to the same end result. On top of that, there are even more possibilities for how you can format the code. This section will take you through some widely used standards for coding to help bring standardisation to this area and make it easier to both write and use our code.\n\n\nClean final code\n\n\n\n\nWhat does this mean?\n\nThis code should meet the best practice standards below (for SQL and R). If you are using a different language, such as Python, then contact us for advice on the best standards to use when writing code.\nThere should be no redundant or duplicated code, even if this has been commented out. It should be removed from the files to prevent confusion further down the line.\nThe only comments left in the code should be those describing the decisions you have made to help other analysts (and future you) to understand your code. More guidance on commenting in code can be found later on this page.\n\nWhy do it?\nClean code is efficient, easy to write, easy to review, and easy to amend for future use. Below are some recommended standards to follow when writing code in SQL and R.\nHow to get started\nWatch the coffee and coding session introducing good code practice below:\n\n\n\n\nThen you should also watch the follow up intermediate session:\n\n\n\n\nClean code should include comments. Comment why you’ve made decisions, don’t comment what you are doing unless it is particularly complex as the code itself describes what you are doing. If in doubt, more comments are better than too few though. Ideally any specific comments or documentation should be alongside the code itself, rather than in separate documents.\n\n\nSQL\n\nFor best practice on writing SQL code, here is a particularly useful word document produced by our Data Hub. This outlines a variety of best practices, ranging from naming conventions, to to formatting your SQL code so that it is easy to follow visually.\n\n\n\nR\n\nWhen using R, it is generally best practice to use R projects as directories for your work.\nThe recommended standard for styling your code in R, is the tidyverse styling, which is fast becoming the global standard. What is even better is that you can automate this using the styler package, which will literally style your code for you at the click of a button, and is well worth a look.\n\n\n\nThere is also plenty of guidance around the internet for best practice when writing efficient R code.\n\n\n\nHTML\n\nIf you ever find yourself writing html, or creating it through rmarkdown, you can check your html using w3’s validator."
+ "objectID": "learning-development/git.html#what-is-git-for",
+ "href": "learning-development/git.html#what-is-git-for",
+ "title": "Git",
+ "section": "What is Git for",
+ "text": "What is Git for\nGit is a version control software that tracks changes to files within a folder that you assign Git to track. It works best with plain text files such as flat data files, code scripts and markdown documents. These folders are known as repositories and can be held and managed securely in a central online place such as GitHub (best for public), GitLab (can be good for either public or private) and Azure DevOps (best for private). We can easily mirror our Azure DevOps repositories in the DfE Analytical Services area on GitHub.\nIt is widely used across DfE and integrates neatly with our use of Azure DevOps, as well as being the current leading version control software in the world of coding with over 87% of 74,298 stack overflow users using it."
},
{
- "objectID": "RAP/rap-statistics.html#peer-reviewing-code",
- "href": "RAP/rap-statistics.html#peer-reviewing-code",
- "title": "RAP for Statistics",
- "section": "Peer reviewing code",
- "text": "Peer reviewing code\nPeer review is an important element of quality assuring our work. We often do it without realising by bouncing ideas off of one another and by getting others to ‘idiot check’ our work. When writing code, ensuring that we get our work formally peer reviewed is particularly important for ensuring it’s quality and value.\nPrior to receiving code for peer review, the author should ensure that all code files are clean, commented appropriately and for larger projects should be held in a repo with an appropriate README file.\nWhen peer reviewing code you should be consider the following questions -\n\nDoes the code do what the author intended?\nIf you’re able to run the code, does it run without errors? If warnings are displayed, are they explained?\nIf the project has unit/integration tests, do they pass?\nAre there any tests / checks that could be added into the code that would help to give greater confidence that it is doing what it is intended to?\nAre there comments explaining why any decisions have been made?\nIs the code written and structured sensibly?\nAre there any ways to make the code more efficient (either in number of lines or raw speed)?\nDoes the code follow best practice for styling and structure?\nAre there any other teams/bits of code you’re aware of that do similar things and would be useful to point the authors towards?\nAt the end of the review, was there any information you needed to ask about that should be made more apparent in the code or documentation?\n\nDepending on your access you may or may not be able to run the code yourself, but there should be enough information within the code and documentation to be able to respond to these questions.\n\n\nReview of code within team\n\n\n\n\nWhat does this mean?\n\nIs someone else in the team able to generate the same outputs?\nHas someone else in the team reviewed the code and given feedback?\nHave you taken on their feedback and improved the code?\n\nWhy do it?\nThere are many benefits to this, for example:\n\nEnsuring consistency across the team\nMinimizing mistakes and their impact\nEnsuring the requirements are met\nImproving code performance\nSharing of techniques and knowledge\n\nHow to get started\nIf you can’t answer yes, then:\n\nGet a member of the team to run the code using only your documentation\nUse their feedback to improve documentation/in-line comments in code\nOther tips for getting started with peer review can be found in the Duck Book\nThe Duck Book also contains some helpful code QA checklists to help get you thinking about what to check\n\n\n\nImproving code performance\n\nPeer reviewing code and not sure where to start? Improving code performance can be a great quick-win for many production teams. There will be cases where code you are reviewing does things in a slightly different way to how you would: profiling the R code with the microbenchmark package is a way to objectively figure out which method is more efficient.\nFor example below, we are testing out case_when, if_else and ifelse.\nmicrobenchmark::microbenchmark( \n case_when(1:1000 < 3 ~ \"low\", TRUE ~ \"high\"), \n if_else(1:1000 < 3, \"low\", \"high\"),\n ifelse(1:1000 < 3, \"low\", \"high\") \n)\nRunning the code outputs a table in the R console, giving profile stats for each expression. Here, it is clear that on average, if_else() is the fastest function for the job.\nUnit: microseconds\n expr min lq mean median uq max neval\n case_when(1:1000 < 3 ~ \"low\", TRUE ~ \"high\") 167.901 206.2510 372.7321 300.2515 420.1005 4187.001 100\n if_else(1:1000 < 3, \"low\", \"high\") 55.301 74.0010 125.8741 103.7015 138.3010 538.201 100\n ifelse(1:1000 < 3, \"low\", \"high\") 266.200 339.4505 466.7650 399.7010 637.6010 851.502 100\n\n\n\n\n\nReview of code from outside the team\n\n\n\n\nWhat does this mean?\n\nHas someone from outside of the team and publication area reviewed the code and given feedback?\nHave you taken on their feedback and improved the code?\n\nWhy do it?\nAll of the benefits you get from peer reviewing within your own team, multiple times over. Having someone external offers new perspectives, holds you to account by breaking down assumptions, and offers far greater opportunity for building capability through knowledge sharing.\nHow to get started\nWhile peer reviewing code within the team is often practical, having external analysts peer review your code can bring a fresh perspective. If you’re interested in this, please contact us, and we can help you to arrange someone external to your team to review your processes. For this to work smoothly, we recommend that your code is easily accessible for other analysts, such as hosted in an Azure DevOps repo and mirrored to github."
+ "objectID": "learning-development/git.html#how-to-install-git",
+ "href": "learning-development/git.html#how-to-install-git",
+ "title": "Git",
+ "section": "How to install Git",
+ "text": "How to install Git\nDownload it from the Git website.\nGit doesn’t have an IDE, instead it will either integrate with your current IDE such as RStudio or Visual Studio Code, or run in the command line.\nWhen you first try to use Git you may be prompted for a GitHub username and password, if this happens you should generate a Personal Access Token (PAT) and use this as your password."
},
{
- "objectID": "RAP/rap-statistics.html#automated-quality-assurance",
- "href": "RAP/rap-statistics.html#automated-quality-assurance",
- "title": "RAP for Statistics",
- "section": "Automated quality assurance",
- "text": "Automated quality assurance\nAny data files that have been created will need to be quality assured. These checks should be automated where possible, so the computer is doing the hard work - saving us time, and to ensure their reliability.\nSome teams are already making great progress with automated QA and realising the benefits of it. The Statistics Development Team are working with these to provide generalised code that teams can use as a starting point for automated QA. The intention is that teams can then run this as a minimum, before then looking to develop more area specific checks to the script and/or continue with current checking processes in tandem. If your team already use, or are working towards using, automated QA then get in touch as we’d be keen to see what you have.\nIt is assumed that when using R, automated scripts will output .html reports that the team can read through to understand their data and identify any issues, and save as a part of their process documentation.\nFor more information on general quality assurance best practice in DfE, see the How to QA guide.\n\n\nBasic automated QA\n\n\n\n\nWhat does this mean?\nThe list of basic automated QA checks, with code examples can be found below and in our GitHub repository:\n\nChecking for minimum, maximum, and average values across your data\nChecking for extreme values and outliers\nEnsuring there are no duplicate rows or duplicate columns\nChecking that where appropriate, geographical subtotals add up to totals (e.g. all the numeric values for LAs in Yorkshire and The Humber add up to the regional total)\nBasic trend analysis using scatter plots, to help you spot outliers and help tell the story of your data.\n\nThe Statistics Development Team have developed the QA app to include some of these basic QA outputs.\nWhy do it?\nQuality is one of the three pillars that our code of practice is built upon. These basic level checks allow us to have confidence that we are accurately processing the data.\nAutomating these checks ensures their accuracy and reliability, as well as being dramatically quicker than doing these manually.\nHow to get started\nTry using our template code snippets to get an idea of how you could automate QA of your own publication files. A recording of our introduction to automated QA is also available at the top of the page.\n\n\n\nPublication specific automated QA\n\n\n\n\nWhat does this mean?\nMany teams will have aspects of their data and processes that require Quality Assuring beyond the generalisable basic checks above. Therefore it is expected that teams develop their own automated QA checks to QA specificities of their publications not covered by the basic checks.\nWhy do it?\nQuality is one of the three pillars that our code of practice is built upon. By building upon the basic checks to develop bespoke QA for our publications, we can increase our confidence in the quality of the processes and outputs that they produce.\nHow to get started\nWe expect that the basic level of automated QA will cover most needs that publication teams have. However, we also expect that each publication will have it’s own quirks that require a more bespoke approach. An example of a publication with it’s own bespoke QA checks will appear in this space shortly. For the time being, try to consider what things you’d usually check as flags that something hasn’t gone right with your data. What are the unique aspects of your publication’s data, and how can you automate checks against them to give you confidence in it’s accuracy and reliability?\nFor those who are interested in starting writing their own QA scripts, it’s worth looking at packages in R such as testthat, including the coffee and coding talk on it by Peter Curtis, as well as this guide on testing by Hadley Wickham.\nThe janitor package in R also has some particularly useful functions, such as clean_names() to automatically clean up your variable names, remove_empty() to remove any completely empty rows and columns, and get_dupes() which retrieves any duplicate rows in your data - this last one is particularly powerful as you can feed it specific columns and see if there’s any duplicate instances of values across those columns."
+ "objectID": "learning-development/git.html#best-places-to-start",
+ "href": "learning-development/git.html#best-places-to-start",
+ "title": "Git",
+ "section": "Best places to start",
+ "text": "Best places to start\n\nContact us about attending one of our in-person Git workshops\nIf you’re new to Git and are unsure of what it does, then take a look through these Git for humans slides\nDavid Sands’ guide to getting started with Git is a helpful place to start.\nGooey Git by David Sands, provides a very neat overview of using git with R."
},
{
- "objectID": "RAP/rap-statistics.html#automating-summary-statistics",
- "href": "RAP/rap-statistics.html#automating-summary-statistics",
- "title": "RAP for Statistics",
- "section": "Automating summary statistics",
- "text": "Automating summary statistics\nAs a part of automating QA, we should also be looking to automate the production of summary statistics alongside the tidy underlying data files, this then provides us with instant insight into the stories underneath the numbers.\n\n\nAutomated summaries\n\n\n\n\nWhat does this mean?\nSummary outputs are automated and used to explore the stories of the data.\nThe Statistics Development Team have developed the QA app to include some of these automated summaries, including minimum, maximum and average summaries for each indicator.\nAt a basic level we want teams to make use of the QA app to explore their data:\n\nHave you used the outputs of the automated QA from the screener to understand the data?\nRun automated QA, ensure that all interesting outputs/trends are reflected in the accompanying text\n\nWhy do it?\nValue is one of the three pillars of our code of practice. Even more specifically it states that ‘Statistics and data should be presented clearly, explained meaningfully and provide authoritative insights that serve the public good.’.\nAs a result, we should be developing automated summaries to help us to better understand the story of the data and be authoritative and rigorous in our telling of it.\nHow to get started\nConsider:\n\nUse the additional tabs available after a data file passes the data screener as a starting point to explore trends across breakdowns and years.\nRunning your publication-specific automated QA, ensuring that all interesting outputs/trends are reflected in the accompanying text\n\n\n\n\nPublication specific automated summaries\n\n\n\n\nWhat does this mean?\n\nHave you gone beyond the outputs of the QA app to consider automating further insights for your publication specifically? E.g. year on year changes for specific measures, comparisons of different characteristics that are of interest to the general public\nAre you using these outputs to write your commentary?\n\nWhy do it?\nAll publications are different, and therefore it is important that for each publication, teams go beyond the basics and produce automated summaries specific to their area.\nHow to get started\nConsider:\n\nIntegrating extra publication-specific QA into the production process\nConsider outputs specific to your publication that would help you to write commentary/draw out interesting analysis"
+ "objectID": "learning-development/git.html#how-to-work-with-git",
+ "href": "learning-development/git.html#how-to-work-with-git",
+ "title": "Git",
+ "section": "How to work with git",
+ "text": "How to work with git\n\nGit Bash\n\nGit Bash allows you to run git commands without opening another IDE. You’d often need to use Git Bash to set your user settings, amend your proxy settings and clone repositories.\n\n\n\nGit with RStudio\n\nGit with R studio is a neat user interface for git. You don’t need to use any git bash commands, and everything is done using point and click. This is useful for day-to-day version control, but does not support the full functionality of git.\nHowever, you can still run the full suite of git commands by simply typing them in the “Terminal” of RStudio."
},
{
- "objectID": "RAP/rap-statistics.html#sensible-folder-and-file-structure",
- "href": "RAP/rap-statistics.html#sensible-folder-and-file-structure",
- "title": "RAP for Statistics",
- "section": "Sensible folder and file structure",
- "text": "Sensible folder and file structure\n\n\n\nWhat does this mean?\nAs a minimum you should have a folder that includes all of the final versions of documents produced and published, per release, within a folder for the wider publication. Ask yourself if it would be easy for someone who isn’t in the team to find specific files, and if not, is there a better way that you could name and structure your folders to make them more intuitive to navigate?\nWhy do it?\nHow you organize and name your files will have a big impact on your ability to find those files later and to understand what they contain. You should be consistent and descriptive in naming and organizing files so that it is obvious where to find specific data and what the files contain.\nHow to get started\nSome questions to help you consider whether your folder structure is sensible are:\n\nAre all documentation, code and outputs for the publication saved in one folder area?\nIs simple version control clearly applied (e.g. having all final files in a folder named “final”?\nAre there sub-folders like ‘code’, ‘documentation’‘, ’outputs’ and ‘final’ to save the relevant working files in?\nAre you keeping a version log up to date with any changes made to files in this final folder?\n\n\n\nNaming conventions\n\nHaving a clear and consistent naming convention for your files is critical. Remember that file names should:\nBe machine readable\n\nAvoid spaces.\nAvoid special characters such as: ~ ! @ # $ % ^ & * ( ) ` ; < > ? , [ ] { } ‘ “.\nBe as short as practicable; overly long names do not work well with all types of software.\n\nBe human readable\n\nBe easy to understand the contents from the name.\n\nPlay well with default ordering\n\nOften (though not always!) you should have numbers first, particularly if your file names include dates.\nFollow the ISO 8601 date standard (YYYYMMDD) to ensure that all of your files stay in chronological order.\nUse leading zeros to left pad numbers and ensure files sort properly, avoiding 1,10,2,3.\n\nIf in doubt, take a look at this presentation, or this naming convention guide by Stanford, for examples reinforcing the above."
+ "objectID": "learning-development/git.html#quick-reference-lookup",
+ "href": "learning-development/git.html#quick-reference-lookup",
+ "title": "Git",
+ "section": "Quick reference lookup",
+ "text": "Quick reference lookup\n\nGitHub have created a cheat sheet for git commands."
},
{
- "objectID": "RAP/rap-statistics.html#documentation",
- "href": "RAP/rap-statistics.html#documentation",
- "title": "RAP for Statistics",
- "section": "Documentation",
- "text": "Documentation\n\n\n\nWhat does this mean?\n\nYou should be annotating as you go, ensuring that every process and decision made is written down. Processes are ideally written with code, and decisions in comments.\nThere should be a README notes file, that clearly details the steps in the process, any dependencies (such as places where access needs to be requested to) and how to carry out the process.\nAny specialist terms should also be defined if required (e.g. The NFTYPE lookup can be found in xxxxx. “NFTYPE” means school type).\n\nWhy do it?\nWhen documenting your processes you should leave nothing to chance, we all have wasted time in the past trying to work out what it was that we had done before, and that time increases even more when we are picking up someone else’s work. Thorough documentation saves us time, and provides a clear audit trail of what we do. This is key for the ‘Reproducible’ part of RAP, our processes must be easily reproducible and clear documentation is fundamental to that.\nHow to get started\nTake a look at your processes and be critical - could another analyst pick them up without you there to help them? If the answer is no (don’t feel ashamed, it will be for many teams) then go through and note down areas that require improvement, so that you can revise them with your team.\nTake a look at the sections below for further guidance on improving your documentation.\n\n\nCommenting in code\n\nWhen writing code, whether that is SQL, R, or something else, make sure you’re commenting as you go. Start off every file by outlining the date, author, purpose, and if applicable, the structure of the file, like this:\n----------------------------------------------------------------------------------------------\n-- Script Name: Section 251 Table A 2019 - s251_tA_2019.sql\n-- Description: Extraction of data from IStore and production of underlying data file\n-- Author: Cam Race\n-- Creation Date: 15/11/2019\n----------------------------------------------------------------------------------------------\n\n----------------------------------------------------------------------------------------------\n--// Process\n-- 1. Extract the data for each available year\n-- 2. Match in extra geographical information\n-- 3. Create aggregations - both categorical and geographical totals\n-- 4. Tidy up and output results\n-- 5. Metadata creation\n----------------------------------------------------------------------------------------------\nCommented lines should begin with – (SQL) or # (R), followed by one space and your comment. Remember that comments should explain the why, not the what.\nIn SQL you can also use /** and **/ to bookend comments over multiple lines.\nIn rmarkdown documents you can bookend comments by using <!-- and -->.\nUse commented lines of - to break up your files into scannable chunks based upon the structure and subheadings, like the R example below:\n# Importing the data -----------------------------------------------------------------------------------\n\nDoing this can visually break up your code into sections that are easy to navigate around. It will also add that section to your outline, which can be used in RStudio using Ctrl-Shift-O. More details on the possibilities for this can be found in the RStudio guidance on folding and sectioning code.\nYou might be thinking that it would be nice if there was software that could help you with documentation, if so, read on, as Git is an incredibly powerful tool that can help us easily and thoroughly document versions of our files. If you’re at the stage where you are developing your own functions and packages in R, then take a look at roxygen2 as well.\n\n\n\nWriting a README file\n\nWhat does this mean?\nA README is a text file (.txt) that introduces and explains a project. It contains information that is required to understand what the project is about and how to use it.\nWhy do it?\nIt’s an easy way to answer questions that your audience will likely have regarding how to install and use your project and also how to collaborate with you.\nHow to get started\nAs a starting point, you should aim to have as many of the following sections as are applicable to your project:\n\nIntroduction\nRequirements (access, software, skills/knowledge)\nHow to use\nHow to contribute\nContact details\n\nThe Self-assessment tool and the QA app give two examples of readme files structured like this."
+ "objectID": "learning-development/git.html#other-resources",
+ "href": "learning-development/git.html#other-resources",
+ "title": "Git",
+ "section": "Other resources",
+ "text": "Other resources\n\nAvison Ho and Linda Bennett gave this coffee and coding presentation on version controlling SQL with Git.\nHappy Git is a useful (though detailed) guide to setting up and using git.\nAdam Robinson and Zach Waller have produced guidance for how to use git in Azure DevOps (formally VSTS), which gives a detailed guide on how to use version control software in DfE analysis.\nWhile also mentioned above as a resource for learning R, chapter 6 of ESFA’s guide to R and Git is also worth looking at for Git alone.\nMicrosoft have produced documentation on using Git within AzureDevOps.\nFor those wanting to go deeper into understand the variety of git commands and what they do, there is a great online visual resource.\nWe also have a number of helpful sections on using git in practice at the end of our RAP page."
},
{
- "objectID": "RAP/rap-statistics.html#version-control-with-git",
- "href": "RAP/rap-statistics.html#version-control-with-git",
- "title": "RAP for Statistics",
- "section": "Version control with git",
- "text": "Version control with git\nIf you do not already have git downloaded, you can download the latest version from their website.\nFor now, take a look at at the resources for learning Git on the learning resources page.\n\n\nVersion controlled final code scripts\n\n\n\n\nWhat does this mean?\nThis means having the final copies of code and documentation saved in a git-controlled Azure DevOps repo in the official-statistics-production area.\nWhy do it?\nHaving the final copy of the scripts version controlled gives assurance around how the data was created. It also allows teams to easily record any last minute changes to the code after the initial final version by using the version control to log this.\nHow to get started\nThe first step is to get your final versions of code and documentation together in a single folder.\nWe have a specific area set up for you to host your publication code in on the dfe-gov-uk instance of Azure DevOps, entitled official-statistics-production.\nTo gain access to this area, please raise a request on service desk by navigating through the pages detailed in the animation below.\n\n\n\nOnce you have navigated to this page, fill out the form with the following details and send your request off.\n\n\n\nAccess is usually granted within a few working days. Alert the Statistics Development Team when this is confirmed, and we will set up your repository and give your team access.\n\n\n\nUse open source repositories\n\n\n\n\nWhat does this mean?\nSaving or cloning your work into a repository that is visible to the public. We currently have brilliant examples of this in our DfE analytical services GitHub area, in which all of the code used to create public dashboards is publicly available.\nFor statistics publications we expect teams to be able to mirror their proccess code on GitHub after publication, which will help open up their code for other analysts to learn from.\nWe are currently working on ways to mirror private repos (i.e. AzureDevOps) to public repos on publication of your data. If you are interested in this please contact the Statistics Development Team.\nWhy do it?\nIt’s a key part of the technology code of practice as an agreed standard for digital services across government.\nHow to get started?\nContact us to get a repository set up in Azure DevOps, to set up a mirroring process to GitHub or to set up a repository on our dfe-analytical-services area on GitHub.\nYou should consider the following principles making an Official Statistics production repository public (some examples are R specific though can be applied to other languages):\n\nFollow the guidance on writing a readme file, and add context in about what Official/National statistics are\nEnsure no data (either input or output) is included in the repository\nHave a clear and organised folder structure (such as having R scripts in an ‘R’ folder)\nCheck your code is styled according the tidyverse styling\nUse renv for package management\nUse an R project\n\nWhen naming your publication’s repository you should use the publication name fully written out, in lower case, and with dashes for spaces – ‘graduate-labour-market-statistics’.\nA single repository should be used for all releases of your publication, there’s no need to have multiple as all the history is saved within previous commits. You can make use of tagging releases in git to help differentiate between each cycle.\n\n\nAvoid revealing sensitive information\n\nHere are some general best practice tips:\n\nUsing .gitignore to ignore files and folders to prevent committing anything sensitive\nNever committing outputs unless they’ve been checked over, even aggregates. We suggest only outputting to an output folder which is in the .gitignore file, to ensure this doesn’t happen by mistake\nKeeping datasets and secrets (e.g. API keys) outside the repository as much as possible, make use of secure variables\nChecking git histories: if someone is planning on open-sourcing code that has previously been in a private repository or only version-controlled locally, you want to be careful not to have anything sensitive in the commit history. You can do this by following the above rules. When in doubt, you can remove the git history and start the public repo without it\nYou can remove a file from the entire commit history if you did commit anything sensitive, although you still need to follow the usual procedures if this was a data breach\n\nYou can find out more in the duck book’s guidance on using git.\n\n\n\n\nCollaboratively develop code using git\n\n\n\n\nWhat does this mean?\n\nHas code development taken place in git, collaboratively across the team?\nAre you making use of pull requests for team members to review and comment on code updates?\nIs there a clear paper trail of changes to code (commits)?\n\nWhy do it?\nUsing git allows multiple people to simultaneously develop the same code using branches, all with a crystal clear audit trail showing what changes were made when using commits. It makes it easy for team members to review changes via pull requests.\nHow to get started\nTo get started you should:\n\n\nGet your code into a git controlled folder\n\nGet code into a git controlled folder in whatever version it is currently in. Use the following steps to do so:\n\nOpen the folder where your project is saved, right click anywhere in that window, and click “Git Bash Here”.\nThis will open a black box. Type in the following and hit enter\n\n\ngit init\n\n\nAfter hitting enter, type in the following and hit enter again after each line. You will need the URL of your Azure DevOps repository to complete this step. Contact the Statistics Development Team if you are not sure what this is or do not have one.\n\n\ngit add .\n\ngit commit -m \"first commit\"\n\ngit remote add origin YOUR_URL_HERE\n\ngit push -f origin --all\n\n\nYou may be prompted for either your windows or git credentials at this stage.\n\nIf prompted for your windows credentials, enter the username and password combination you use to log into your DfE device.\nIf prompted for your git credentials, visit your online repository, click on the blue “clone” box, and click “generate git credentials”. This will generate a username and password for you to enter.\n\n\n\nVisit your repository online, and check that all the files have uploaded. Other members of your team will now be able to work from your code.\n\n\n\n\nBuild capability within the team\n\n\nEnsure all team members have access to your project in the Azure DevOps official-statistics-production area. Contact the Statistics Development Team if there are any issues.\nGet team members to clone your repository in to their personal area, so everyone is able to work on code at the same time.\n\nTo clone code, they will need to do the following:\n\nRun through steps 1 - 2a of getting a file into a git controlled folder\nAfter running those lines, type in the following with your repository URL in the “YOUR_URL_HERE” space. This will clone the online repository to your local area.\n\n\ngit clone YOUR_URL_HERE\n\n\nMake use of git and version control in your team projects regularly. Like learning anything new, putting it into practice regularly is the best way to become confident in using it.\n\nPlease refer to the other links on the Git learning resources page to learn more about how to use git in practice."
+ "objectID": "learning-development/git.html#tips-for-using-git",
+ "href": "learning-development/git.html#tips-for-using-git",
+ "title": "Git",
+ "section": "Tips for using Git",
+ "text": "Tips for using Git\n\nBranches\n\nDavid Sands has produced a very helpful video on how to use branches in git, which also covers how to tackle merge conflicts if and when they arise:\n\n\n\n\nWhen you create a git project, it will automatically create a “main” (sometimes “master”) branch for you. This is where code that has been QA’d and you are happy with should sit.\n\nIt is good practice to have at least one other branch, we tend to call it “development”. This is the branch where you will be doing most of your work. To open a new branch, navigate to “branches” and click on the blue box highlighted below\n\n\n\nHaving two separate branches means that if anything goes wrong in the “development” branch, the “main” or “master” branch is still unaffected and runs without issue. This lets you test and QA the code more thoroughly before merging into your main branch.\nWhen working on your project, make sure that you are in the right branch. You can check this by navigating to the “Git” tab in RStudio as demonstrated below.\n\n\n\n\n\nCommits\n\nOnce you are happy with changes and want them to be in the latest version of your branch for all of your team to see, you can push “commits” up.\n\nWhen you make a change to a file, this will pop up in the “Git” window of your R console. Select the files you want to commit by ticking the “staged” box next to them.\n\n\n\nThis will bring up a new window. Add a comment describing your additions/changes, and click commit. You will see all the staged files disappear. Then click “Push” to push the committed files up to the online repository for all to use.\n\n\n\nWhen another member of your team makes a commit and you want to pull this into your local area to check and work off the latest version, click on the blue “pull” button.\n\n\nCommitting regularly is strongly recommended as each commit is a saved point in time that you can easily roll back to if needed. If you want to know more about how to do this, see the reverting a commit section below.\n\n\n\nPull requests\n\nWhen you have got to a place with the code and your committed changes where you are happy for it to be QA’d, you can open a pull request. This gives your team a chance to QA your changes before merging the branches together.\n\nNavigate to “Pull Requests” in the “Repos” tab of Azure DevOps and click the blue “New pull request” button.\nThis will take you to a new window. Here, you can add:\n\nTitle, tell your team what has changed\nDescription, tell your reviewer what they should check\nReviewers, add multiple if needed.\n\nAs a reviewer, to approve a pull request, follow the link in your email and click “approve” in the blue box. When all reviewers are happy for this to be the new master branch, click “complete”.\n\nCreating pull requests in GitHub follows a broadly similar process and should be intuitive from the above steps for Azure DevOps.\n\n\n\nMerging a branch into another one\n\nIf you want to merge a branch into another without doing a full pull request, you can do this using a terminal.\nThis may happen if you are working on a feature branch, and want to merge the latest changes to the master branch back into your feature branch before opening a PR to master with your own changes. Often this can let you deal with nasty conflicts up front, or allow you to keep working on your feature but update to have something new that’s on master.\nStart off by checking out your desired target branch - git checkout mybranch. Then merge in the branch you want (e.g. master) - git merge master.\n\n\n\nCherry picking\n\nIf you want to cherry pick specific commits for PR, you can do this by cherry picking the commits you want to use, and creating a new branch that has only those new commits that you want.\nTo start off you’ll need to identify the commits you want. In the terminal, run git log --oneline to get a log of commits for your current branch, use git log --online BRANCHNAME to specific the branch for the log. This gives a list of commit hashes and messages (stackoverflow response defining git hashes and commmit ID’s). You can press enter to get more commits, or q to quit\nThen go to the branch you want the commit to appear on and cherry pick your commits. Often, if this is to hop around something on a development to master pr, you would create a second development branch, cherry pick commits to there, and then PR that to master and delete the branch after merging.\nOn the branch you want to PR (i.e. your copy of a development branch purely for merging only cherry picked commits) run git cherry-pick COMMITHASH to add the specified commit, or git cherry-pick HASH1^..HASH4 for a specified list of commits (inclusive).\nHappy days!\n\n\n\nVisualising your tree\n\nYou can use gitk --all to visualise a tree of all previous commits up to this point.\n\n\n\n\nGetting commit IDs\n\nCommit IDs are the way Git identifies unique commits. They’re really helpful if you ever need to revert back to a previous commit if you’ve made a mistake.\nThere are lots of different ways to find out commit IDs:\n\nVisit the repo in Azure Devops and go to the commit of interest. At the top of the page there is a commit ID you can copy.\nNavigate to the repo in your file explorer, then open up Git Bash and type\n\ngit log --pretty=format:\"%h - %an, %ar : %\nOR\ngit log --pretty=oneline\nThere are a number of customisable versions of this, more information is available on the Git website.\n\n\n\nReverting a commit\n\nMade a mistake and need to revert? No problem! Reverting commits in git creates a new commit reversing your accidental commits, bringing you back to an earlier point in your branch.\nFor rolling back on a branch, you should revert any changes so that you’re not erasing history others might have pulled/cloned. E.g. If you want to revert the last 3 commits on master:\ngit revert --no-commit master~3..master\nThe no commit argument means that it just makes the reverts locally, and you can then commit them all as a single revert commit. If you don’t use --no-commit then it will start doing individual revert commits for you for each commit you’re reverting.\nYou can also revert back to a particular commit ID:\n\nNavigate to the repo in your file explorer, then open up Git Bash and type :\n\ngit revert [PASTE COMMIT ID HERE]\n\nThis opens up a window that asks you to write a commit message. You can skip this step as it automatically writes a revert message for you. Enter :wq which quits the writer window.\nYou can now push these changes to Azure Devops by entering git push into the git window\n\n\n\n\nTagging release versions\n\n\nIt can be useful to tag specific commits or releases at key points in time. For us a common example will be each publication cycle, to tag the version of the code used to process data for a particular release or amendment.\nGuidance on how to tag releases using git can be found on the draft version of the BPI code guidance\n\n\n\nFix: cannot resolve proxy\n\nIf you get this error when trying to pull / push to a repository from a DfE laptop:\nfatal: unable to access 'https://dfe-gov-uk.visualstudio.com/stats-development/_git/ees-analytics-new/': Could not resolve proxy: mwg.proxy.ad.hq.dept\nThen try running the following git command to clear the proxy settings in your git config file:\ngit config --global --unset http.proxy\n\n\n\nCleaning up local branches\n\nEssentially this finds all merged (old) branches, makes sure to not include master or development (can rename or add more if appropriate) and then runs those branches through the delete command. To be safe, run the first two thirds first, to print out the list of what you’ll be deleting:\ngit branch --merged | egrep -v \"(^\\*|master|development)\"\nThen if you’re happy, run the whole line:\ngit branch --merged | egrep -v \"(^\\*|master|development)\" | xargs git branch -d\nThat should be all local branches tidied up. Now to complete the job you can prune the tracking branches you have set up, usually this will just be:\ngit remote prune origin\nThough you can use git remote -v to find other remotes if you have them.\n\n\n\nCreating PATs\n\nPATs or Personal access tokens, are a preferred way to authenticate into repositories through code. Creating a PAT in GitHub is relatively intuitive.\nIn Azure DevOps, you can do this by following Microsoft’s documentation on creating a PAT in Azure DevOps.\n\n\n\nStoring secure variables\n\n\nAzure DevOps\n\nIn Azure DevOps you can securely store variables that are then used by your pipelines by making use of variable groups.\nFirst create a variable group by navigating to Pipelines > Library. Enter any variables you want to store here and make sure to change the variable type to secret if appropriate (i.e. login credentials or PATs).\n\nThen, in the pipeline you want to use the variables in, go to Variables > Variable groups and link the variable group as shown below. You can then call upon the variables as needed in your pipeline.\n\n\n\n\nGitHub\n\nIn GitHub you can store sensitive variables as encrypted secrets.\n\n\n\n\nMirroring a repository\n\nThere may be times when you have a repository in one place, such as Azure DevOps, but want to mirror the code and any changes to it in another place, such as GitHub. This can be for many different reasons, though commonly for us it will be to open up our code across government. Azure DevOps doesn’t provide us with easily publicly visible code repositories, however GitHub does.\nTo mirror a code repository from Azure DevOps, use the mirror git repository extension (already installed on dfe-gov-uk instance), make use of the PAT and secure variables sections above and then add a job to your pipeline and enter your parameters in the fields as per the example below:\n\n\n\n\nFixing the “JSON token 4” error\n\nThis error can appear when you are trying to push changes to Azure DevOps.\nIf it appears and you have not changed your password recently, try locking your laptop and seeing if on re-login it prompts you for a password change.\nIf you have changed changed your password recently, try signing out of Azure and then back in again."
},
{
- "objectID": "statistics-production/ees.html",
- "href": "statistics-production/ees.html",
- "title": "Explore Education Statistics",
+ "objectID": "learning-development/learning-support.html",
+ "href": "learning-development/learning-support.html",
+ "title": "Learning support",
"section": "",
- "text": "Guidance for how to use the features in the Explore Education Statistics platform\nExplore Education Statistics (EES) is the Department’s official statistics dissemination platform, designed to make DFE’s published statistics and data easier to find, access, use and understand.\nThe EES platform consists of two applications:\nBoth applications were released to Minimal Viable Product (MVP) standard in March 2020, we will be iterating and improving the functionality based on user feedback throughout 2020 and 2021.\nMaintenance and use of the platform is supported by the Statistics Development Team."
+ "text": "A collection of useful learning resources, and information on support and resources to get you started\nAs statistics producers we require a variety of tools to be able to process our data. Below are the recommended tools that will give us the most power to do what we need. These have large user communities in DfE, and are already working in our current IT setup.\nFor best practice when using software and coding in our process, see our guidance on RAP expectations the DfE Good Code Practice guide."
},
{
- "objectID": "statistics-production/ees.html#environments",
- "href": "statistics-production/ees.html#environments",
- "title": "Explore Education Statistics",
- "section": "Environments",
- "text": "Environments\nAs stated above, EES consists of two parts. We also have four versions (environments) of EES, the banner for the admin part of each environment will inform you which environment you are on, this is also colour coded:\n\nDevelopment - Green - where changes are first merged in, and often the first time different pieces of work from different developers will interact properly\nTest - Pink - where our developers can carry out manual testing of any new features to make sure things work as expected\nPre-production - Yellow - a sandbox area for analysts to carry out functionality testing\n\nAdmin pre-production - https://admin.pre-production.explore-education-statistics.service.gov.uk\nPublic pre-production - https://pre-production.explore-education-statistics.service.gov.uk\n\nProduction - Red - the real service, anyone creating real releases that they intend to publish to the public should be using this environment\n\nAdmin production - https://admin.explore-education-statistics.service.gov.uk\nPublic production - https://explore-education-statistics.service.gov.uk\nThere is no overlap between the environments and content created on one cannot be moved to any other.\n\nIf you are bookmarking links, please be careful to bookmark the links below exactly as they are shown. Often when signing in you will be redirected via other url’s as a part of the authentication process, and bookmarking those may lead to errors.\n\nTeams can use the pre-production environment to familiarise themselves with the platform and test out what is possible. The core functionality across the environments is identical except for new changes, which are deployed through the different environments before they make it to production.\nOn the pre-production environment, all analysts have full permissions to create publications and releases, and can see everything else that other analysts are making. This is unique to the pre-production environment, on the production environment analysts will only have access to releases that they have been granted specific access to.\nWhen accessing the pre-production environment you may be asked for a username and password, these are as follows: dfe, dataresearch.\n\nThe pre-production environment is not suitable for unpublished data. Unpublished data should only be uploaded to the production environment.\n\n\nGetting access to admin\nAccess to the production and pre-production admin services is limited to DfE AD accounts only and users have to have been invited to the service by either the Statistics Development Team (for full access) or an existing user (for pre-release access). Invites to the service are sent out via email using gov.uk Notify.\nTo be invited to the service for full access teams need to email the Explore Statistics Mailbox, stating who needs access, what permissions they require (analyst or approver), and for which publications these apply. This email should be sent by the Team Leader, or accompanied with the relevant Team Leader’s approval. Once access has been granted you will receive an email inviting you to use the platform.\n\nJobshare emails can not be used to access EES. Specific personal emails should be used instead, in the same way as you sign in to windows on your machine."
+ "objectID": "learning-development/learning-support.html#email-support",
+ "href": "learning-development/learning-support.html#email-support",
+ "title": "Learning support",
+ "section": "Email support",
+ "text": "Email support\n\nstatistics.development@education.gov.uk\n\n9-5, aim to reply between 1-2 days."
},
{
- "objectID": "statistics-production/ees.html#roles-and-permissions",
- "href": "statistics-production/ees.html#roles-and-permissions",
- "title": "Explore Education Statistics",
- "section": "Roles and permissions",
- "text": "Roles and permissions\nThe Statistics Development Team are responsible for setting up and maintaining user permissions during the beta phase. Change requests will be monitored via their main mailbox - explore.statistics@education.gov.uk.\nThe following roles exist within EES admin and are assigned to a specific release:\n\nPre-release\nAny user invited for pre-release is given a pre-release user role. During the hours of pre-release they can:\n\nPreview the release page, including downloading files\nPreview the table tool page for that release\n\n\n\n\nAnalyst - contributor\nAny analyst working on a release within EES admin will have the contributor role. They can:\n\nEdit release details\nUpload data and other files\nCreate footnotes, datablocks and charts\nEdit and comment on release content\nInvite PRA users\nMove the release status between draft and ready for higher review\n\n\n\n\nAnalyst - approver\nThe responsible statistician for a statistics release will have the approver role, allowing them to sign off releases for publication. This will usually be the responsible G5 or G6 for the statistics publication. They can do everything a contributor can, as well as:\n\nMove the release status between draft, ready for review and approved\nSchedule the release for publication (either immediately or for a specific date)\n\n\n\nThere is also a publication owner role, which is assigned at publication level to a users account:\n\nPublication owner\nThis permissions level gives publication owners control over their publications. They can do anything a contributor can for all releases within the publication. On top of this the publication owner also has access to:\n\nManage publication level details\nCreate new releases within a publication\nCreate an amendment of the latest published release within a publication\nCancel an amendment before it is published\n\n\n\n\nBAU\nThe administrative role:\n\nThis role is assigned to a user and gives full access including administrative tools, typically only used by the Statistics Development Team and EES developers.\n\n\nSee below for a diagram of the responsibilities of each role as part of the publication process in EES:"
+ "objectID": "learning-development/learning-support.html#technical-workshops",
+ "href": "learning-development/learning-support.html#technical-workshops",
+ "title": "Learning support",
+ "section": "Technical workshops",
+ "text": "Technical workshops\n\nIn person workshops covering specific technical skills in practice\n3 hours long, with people working in small groups\nRecently circled the sites running an introduction to Git, and introduction to R and RAP.\nContact statistics.development@education.gov.uk to register interest in workshops happening at your site or to request new topics!"
},
{
- "objectID": "statistics-production/ees.html#requesting-a-new-publication",
- "href": "statistics-production/ees.html#requesting-a-new-publication",
- "title": "Explore Education Statistics",
- "section": "Requesting a new publication",
- "text": "Requesting a new publication\nIf you have not published on EES before, or if you’re creating a brand new publication, you will need to contact us to create the publication for you before you can get started.\nThe hierarchy of content within EES is as follows -\n\nPublications are organised into topics and themes (as shown on the EES Find Statistics page), and then within each publication there are releases - where the latest release includes the latest statistics for that publication. For example:\n\n\n\nLevel\nExample\n\n\n\n\nTheme\nPupils and Schools\n\n\nTopic\nPupil absence\n\n\nPublication\nPupil absence in schools in England\n\n\nRelease\n2018/19\n\n\n\nEach publication should have a lead statistician’s name and contact details attached to it as a requirement of the Code of Practice. You may use a team email address but the name and telephone number of a named statistician should also be provided.\nMethodology documentation is also attached at a publication level within EES - meaning one standalone piece should be written to cover all releases for the given publication within the service.\nIf you want to request a new publication, please be prepared to give the following:\n\nThe heading and subheading that you want the publication to sit under in our site (take a look at the current structure here), feel free to either use existing ones, or suggest new ones\nThe exact title of the publication\nA summary of less than 160 characters (including spaces) that describes your publication series\nIf the methodology already exists elsewhere and you have a URL for it, or if you want to create one\nTeam name\nTeam email address\nLead statistician name\nLead statistician contact number\nThe time period of the release you’re creating\nWhether they are National statistics, an ad-hoc publication, or official statistics\nList of email addresses of analysts who need access to the release and at what permission levels\n\nPublication details can be managed by publication owners or via requests to the Explore Statistics Mailbox."
+ "objectID": "learning-development/learning-support.html#partnership-programme",
+ "href": "learning-development/learning-support.html#partnership-programme",
+ "title": "Learning support",
+ "section": "Partnership programme",
+ "text": "Partnership programme\n\nDedicated resource to support development work, either building skills or working on statistics outputs.\nA short-term project with an agreed set of goals and timescale decided as part of an initiation meeting.\nContact statistics.development@education.gov.uk if you’re interested in finding out more about the support we can give."
},
{
- "objectID": "statistics-production/ees.html#admin-dashboard",
- "href": "statistics-production/ees.html#admin-dashboard",
- "title": "Explore Education Statistics",
- "section": "Admin dashboard",
- "text": "Admin dashboard\n\nUse Google Chrome or Microsoft Edge to access and use the admin part of Explore Education Statistics.\n\nWhen you enter the service you’ll see the admin dashboard, here what you can see will be dependent on your access permissions i.e. you’ll only see the publications that you have been granted access to. Within the test environment users are set up with full permissions (i.e. they can see/do everything) though in our production environment this will be restricted.\nThe Statistics Development Team will be responsible for setting up and maintaining user permissions during the beta phase. Change requests will be monitored via their EES mailbox - explore.statistics@education.gov.uk.\n\nWithin this dashboard you can view and manage existing publications, including creating and editing their releases. You can use the drop down lists to find releases by theme/topic/publication or use the draft and scheduled releases tabs to see releases that are in progress."
+ "objectID": "learning-development/learning-support.html#specific-resources",
+ "href": "learning-development/learning-support.html#specific-resources",
+ "title": "Learning support",
+ "section": "Specific resources",
+ "text": "Specific resources\nWe also have specific learning resources and materials for SQL, R, and Git"
},
{
- "objectID": "statistics-production/ees.html#external-user-access",
- "href": "statistics-production/ees.html#external-user-access",
- "title": "Explore Education Statistics",
- "section": "External user access",
- "text": "External user access\nAccess to EES for users outside of DfE is difficult as the authentication and security on the service relies on the DfE Azure Active Directory and associated infrastructure. It is possible to request that external users from other government organisations have access, and they can then be added as collaborating analysts on a release or as pre-release viewers.\nIf you have external users you’d like to request access for, please send the following to explore.statistics@education.gov.uk, at least two weeks in advance of requiring access. Due to the dependency on DfE’s digital security we cannot guarantee access or how long it may take.\n\nEmail addresses of users to be added\nReason for access\nLength of time access is needed"
+ "objectID": "statistics-production/stats_tools.html",
+ "href": "statistics-production/stats_tools.html",
+ "title": "Tools for statistics",
+ "section": "",
+ "text": "Guidance for how to use the tools available for statistics publishers\n\n\nWe also have a number of tools for statistics production teams to use to help them in their processes, they can be accessed at the links below when using a DfE device.\n\nEES data screening app\nThis application allows you to screen your data files against the underlying data standards for Official and National statistical publications. If your file passes the checks it also gives you some basic options for exploring the data in the file.\n\nEES data screener\nData screener code on GitHub\n\n\n\nEES analytics\nThere is an R Shiny dashboard that provides access to the Google Analytics data for the Explore Education Statistics platform.\n\nExplore Education Statistics analytics dashboard\nCode is not yet publicly available, it is stored in Azure DevOps, contact us if you are interested in seeing it or contributing.\n\n\n\nPublication RAP self-assessment tool\nA self assessment tool allowing official statistics publications to understand how their production processes match up against RAP expectations.\n\nRAP self-assessment tool\nRAP self-assessment code mirrored on GitHub\n\n\n\nTemplate code\n\nTemplate DfE R Shiny dashboard\nTemplate QA code repository\ndfeR R package\n\n\n\n\n\n Back to top"
},
{
- "objectID": "statistics-production/ees.html#data-and-files",
- "href": "statistics-production/ees.html#data-and-files",
- "title": "Explore Education Statistics",
- "section": "Data and files",
- "text": "Data and files\nHere you will update your data files and accompanying metadata. You need to make sure that the data has passed through our data screener checks before trying to upload it.\n\nAll data files are accessible in the downloadable files for users to explore in the same format as they are uploaded.\n\n\n\nSubject titles\n\nYou’ll need to give a ‘subject title’ to each data file you upload. The subject name should be a simple user-friendly title for the data included within the file - it is what users will see wherever the file is referenced within EES. The actual file name and data guidance can include more technical / coverage information.\nWhen adding a subject name think about the general user and how the subjects will appear in the service:\n\nYou don’t need to include the publication name in the subject name as this is always already implied within EES\nYou don’t need to list what filters are in each file in the title, users can see this in the data guidance\nYou don’t need to include the date ranges covered in each file in the title, users can see this in the data guidance\nYour subject names should be short and snappy, clearly explain what is in each file. Some good examples of this in practice are included below:\n\nEarly years provision by provider type\nExclusions and suspensions by pupil characteristics\nITT new entrants by subject and training route\n\n\nYou can make changes the subject title for your data file after it has been uploaded if needed using the ‘Edit title’ option.\n\n\n\nUploading files\n\nWhen uploading files you have a choice between uploading as separate CSV files or as a combined ZIP file.\nFor data files greater than 80mb we recommend uploading as a ZIP file.\nOnce you click to upload the file a ‘Status’ will be visible that shows the progress of the import process. It will go through the following stages:\n\nQueued\nValidating\nImporting\nComplete\n\nThis may take a little while depending on the size of your file and if there are numerous files queued for import. You cannot view the dataset or use it to create tables/charts until this status is ‘COMPLETE’.\n\nIf you believe your file is ‘stuck’ please contact explore.statistics@education.gov.uk, with details of the file and the release that you are uploading to.\n\n\n\n\nOrdering filters and indicators\n\nTo save time when creating tables and charts, and to aid users who view your data themselves via the table tool, you can now save custom orders for your filters and indicators. You can do this in the data and files page after uploading underlying data files, via the ‘reorder filters and indicators’ tab. Once there just choose which file you want to reorder and then click and drag the items until they’re in the order you want them to show in the table tool. This then becomes the default order for this filter or indicator and will apply in all charts and tables automatically.\n\n\n\n\nAncillary file uploads\n\nAny files you want to make available for users to download but aren’t intended the table tool should be added as an ancillary file upload. These files will need to meet all requirements of the new accesibility regulations before they can be published.\nExamples of ancillary files may be:\n\nInfographic pages\nSupplementary data that isn’t intended for the table tool\nAdditional technical documentation\n\nFor ensuring that spreadsheets are accessible see the guidance from .gov.wales, and this blog post that walks you through some ways that you can recreate the experience of using spreadsheets.\n\nAny data files that only consist of Planning area, and/or Institution level data should be uploaded as an ancillary file, rather than as a data file.\n\n\n\n\nPublic data guidance\n\nHere you should also create your public data guidance document, this replaces the information that would have previously been uploaded as a pdf and is designed to help users understand and use the data they download from your release.\n\nThe document will automatically update as you add new data files to your release, however you will need to add an overview of all the data included in the release and short summaries for each data file before the release can be published.\nA list of variables in each file with an associated label (taken from metadata uploads) and associated footnotes will also be displayed for each file.\n\n\nReleases cannot be published without a completed metadata document. If not filled in an error will be flagged during sign off.\n\n\n\n\nReplacing data\n\nIf you just need to change the subject title for your data file you do not need to go through the whole replacement process, just click the ‘Edit title’ option.\nHowever, if you notice a mistake in your data file you can replace it with another. When replacing a data file the platform will attempt to recreate any data blocks and footnotes that were built using the previous file.\n\nThe replacement file must contain the exact same column names and types as the original. For example, a character column named “date” must also be replaced with a character column named “date”. A numeric column named “date” will not work in the replacement.\n\n\nThe first step is to upload the new file.\n\nOne you’ve chosen and uploaded your replacement file it will need to go through the usual import process before it can check if retaining existing data blocks and footnotes will be possible.\n\nOnce the upload is finished a report will appear which highlights whether existing data blocks and footnotes can be replaced successfully. If you want to keep any data blocks and footnotes you’ve built you will need to make sure that your replacement data file still contains the information (indicators, filters, geographic_levels and time_periods) that was used to create them.\n\nIf it’s not possible for a data block or footnote to be recreated using the replacing data file a warning will appear and you’ll be prompted to either edit or delete them before completing the replacement.\n\n\nRemember to double check any data blocks or footnotes that were recreated by the platform before publishing your release."
+ "objectID": "statistics-production/user-eng.html",
+ "href": "statistics-production/user-eng.html",
+ "title": "User engagement",
+ "section": "",
+ "text": "Guidance on understanding and engaging with the users of published statistics"
},
{
- "objectID": "statistics-production/ees.html#footnotes",
- "href": "statistics-production/ees.html#footnotes",
- "title": "Explore Education Statistics",
- "section": "Footnotes",
- "text": "Footnotes\nWe generally advise against using footnotes, unless they’re absolutely necessary (think about how many people will actually be reading them!). Consider where you could signpost to the methodology /data guidance / commentary instead. However, if you do want to add footnotes to your data, this should be done using the EES footnote function.\nFootnotes are added via footnotes tab - rather than writing multiple tables and assigning individual footnotes, you write footnotes and assign them to certain indicators and filters so they appear when users select them in the table builder. For example in the below, the footnote “This is a footnote” is assigned to “Headcount” indicator for all options within the “School type” filter.\n\nIf you’d rather, you can assign a footnote to the whole data file by ticking this box.\n\nYou can assign the same footnote across multiple data files.\n\nWe recommend that you only add footnotes once you are certain the data file is final. If you have to delete the data file, all the assigned footnotes will be deleted alongside it."
+ "objectID": "statistics-production/user-eng.html#user-hub",
+ "href": "statistics-production/user-eng.html#user-hub",
+ "title": "User engagement",
+ "section": "USER hub",
+ "text": "USER hub\nThe USER hub is now available on the GSS website, this is a central resource to help producers of statistics develop the knowledge, skills and techniques needed to engage effectively with their audiences."
},
{
- "objectID": "statistics-production/ees.html#data-blocks-tables-and-charts",
- "href": "statistics-production/ees.html#data-blocks-tables-and-charts",
- "title": "Explore Education Statistics",
- "section": "Data blocks (tables and charts)",
- "text": "Data blocks (tables and charts)\nA data block is a smaller cut of data from your original file that you can embed into your publication as a presentation table, build charts from, or link users directly to.\n\n\nTables\n\nHere you can create data blocks, using the admin table tool to create and save your summary tables. Remember to use the reorder table headers tool to restructure your table however you want it before you save.\nOnce you’re happy with your table, give it a title and some source information before clicking save. The source should be the source of the data used to create the data file that was then used for the data block table, for example ‘School Census’.\nThe manage data blocks page will list all the data blocks you have created, highlighting which have charts, are used in content and are saved as highlight tables.\n\n\n\nFeatured tables\n\nYou can also choose to highlight a data block table as a ‘featured table’ which means it will show in a list of featured tables within the table tool. This is designed to help users get to tables of interest more quickly (without having to create tables themselves).\nThere is an option to choose if a table is a ‘featured table’ when saving each data block, here you can name the table and add a description giving the table coverage:\n\nEach featured table will then be listed to the user within the table tool. Featured tables do not have to be embedded within your release content to be included in this list.\n\n\n\n\nFast track links\n\nAny data block tables that are created and saved within the admin app will also be assigned a ‘fast-track link’, this url can then be used throughout your release as a way to direct users to specific tables (within the table tool) more quickly so they can interact and explore the data further. It will appear at the top of the page like this:\n\nWhen your release is published, any embedded data block tables within the release will have an ‘explore data button’ beneath them which will utilise these fast track links to quickly direct users to the table within the table tool so they can explore the data further. You can also use fast-track links as a hyperlink within release commentary (without having to embed the data block).\nIn your fast-track titles, you don’t want to overload information, but still want to direct the user to the right place. Remember they can go back to the table tool through your featured tables to change filters and indicators as needed.\nFast-track titles should explain:\n\nWhat the table is showing in the simplest terms\nWho/Where the data covers (e.g. characteristic groups and geography levels)\nWhen the data in the table is reported for\n\nHere are a few examples of good fast-track titles in EES:\n\nNumber of Schools and Pupils, by School Type, 2015/16 to 2020/21\nAbsence Rates by School Type, 2016/17 to 2020/21\nFree School Meals, by Region, 2015/16 to 2020/21\n\n\nWhat is the difference between a fast-track link and a permalink?\nEES also offers ‘permalinks’ for any table created in the table tool which allows a user to save a link to a permanent, static, version of a table they have created. Analysts can make use of these permalinks when answering queries or in PQ and FOI responses.\nFast-track links are similar to permalinks however instead of linking to a static version of a table they link to an ‘active’ version of the table within the table tool - meaning users can interact and change what’s shown in the table from within the table tool if they choose to.\n\n\n\n\n\nCharts\n\nAfter building and saving a data block table you will see a ‘Chart’ tab appears. This tab will take you to the EES chart builder, where you can choose to add a chart to your data block.\nThe first step to creating a chart is choosing the chart type, currently the EES chart builder can build line charts, horizontal/vertical bar charts (including stacked and clustered) and maps.\nAfter choosing your chart type you then need to work through the following stages to build your chart. In each stage you’ll be shown a live preview as you make changes.\n\n\nChart configuration\n\nWithin the ‘chart configuration’ tab you can add a title, alt text, move the legend and change chart dimensions.\n\nMake sure to review your chart dimensions before you publish. Users should be able to read the labels on the axes and see the legend without having to scroll.\n\n\nNote, within the vertical and horizontal bar chart types you can also create stacked bar charts by clicking the ‘Stacked bars’ option within the chart configuration tab\n\n\n\nData sets\n\nHere is where you add data to the chart. You can add each series one at a time or all together.\n\n\n\n\nLegend\n\nYou can edit the chart legend, and styling of your series via the Legend tab.\n\n\nTo select specify custom colours outside of the defaults, you can double click on the colour codes at the bottom of the colour picked until you get to the type of code you’re wanting to input (e.g. hex code) and then enter the code manually.\n\n\n\n\nX axis (major axis)\n\nHere is where you configure the x-axis: You can alter gridlines, labels, sort, limit and add reference lines.\n\n\n\n\n\n\n\n\nOption\nWhat it does\n\n\n\n\nSize of axis\nChange the width of the space given to axis tick labels\n\n\nShow grid lines\nTurn grid lines on and off\n\n\nShow axis\nTurn the axis on and off, you can also add a unit to the axis tick labels\n\n\nSorting\nChange how the data within the chart is sorted\n\n\nTick display type\nAlter how often axis tick labels are shown, labels will automatically skip values where there are too many to show without overlapping\n\n\nAxis range\nAlter the range of data shown in the chart\n\n\nLabel\nAdd an axis label, you can also choose the width for the space given to it\n\n\nReference lines\nAdd/remove reference lines to the chart\n\n\n\n\n\n\nY axis (minor axis)\n\nThen follow a similar process for the y axis configuration, play around until the chart looks how you want it to.\n\n\n\n\n\n\n\n\nOption\nWhat it does\n\n\n\n\nSize of axis\nChange the width of the space given to axis tick labels\n\n\nGroup data by\nChange how the data within the chart is grouped\n\n\nShow grid lines\nTurn grid lines on and off\n\n\nShow axis\nTurn the axis on and off, you can also add a unit to the axis tick labels\n\n\nSorting\nChange how the data within the chart is sorted\n\n\nTick display type\nAlter how often axis tick labels are shown, labels will automatically skip values where there are too many to show without overlapping\n\n\nAxis range\nAlter the range of data shown in the chart\n\n\nLabel\nAdd an axis label, you can also choose the width for the space given to it\n\n\nReference lines\nAdd/remove reference lines to the chart\n\n\n\n\n\n\nChanging chart type\n\nIf you create your chart and then change your mind as to what chart type would be best you can just click to change it and it will try to save all the options that you had applied previously.\n\n\nRemember to save your chart when you’re done.\n\n\n\n\nMaps\n\nYou can create maps too, currently this is possible for regional, LA and LAD data.\nYou can change the boundaries you are plotting onto via the “chart configuration” tab, the latest boundary file will automatically be selected, but if you are visualising historic data, you may want an older boundary file. Please contact us if the boundary you want to plot is unavailable.\n\nThen to create a map, add the cuts of data you want to display in the “data sets” tab of the chart builder.\n\nYou can change the colour scale of the chart in the “legend” tab.\n\n\n\n\nOther chart types\n\nYou should only use complex charts where there is a clear user need, as simple charts are the easiest for users to understand. If you have a an infographic or a chart that is too complex to build within our chart builder you can use the ‘Choose an infographic as alternative’ option to upload an image to your data block or use the Add embed block feature to embed an R-Shiny based plot (see the section on embedded blocks for further information).\nInfographic alternatives should be .svg format and you can use the sizing options within the data block editor to size your image appropriately.\n\nAccessibility matters for infographics too, consider the following if you do need to upload an image -\n\nKeep them simple\nUse colours that are available in EES - see our visualisation guidance for more details\nDon’t put borders around images\nRead the GSS guidance on the use of colour in visualisations and follow the steps provided to check your visualisations\nTry to avoid adding text to your images other than axis labels and limited annotations. Titles and headings can be added in the “chart configuration” tab instead\n\n\nYou can use R to create infographics and the following code gives an example of how to create a basic line chart or stacked bar chart using the appropriate GSS sequential colour palette.\n# Load the necessary libraries\nlibrary(ggplot2)\nlibrary(dplyr)\n\n# This is the GSS standard categorical colour palette\ngss_categorical_palette <- data.frame(\n names = c('Dark Blue', 'Turquoise', 'Dark pink', 'Orange', 'Dark grey', 'Light purple'),\n hex = c(\"#12436D\", '#28A197', \"#801650\", \"#F46A25\", \"#3D3D3D\", \"#A285D1\"),\n id = c(1,2,3,4,5,6))\n\n# Set up a dummy data-set.\n# Note the line factor(...,levels=...) allows you to order your filter values in\n# the final plot based on the ordering entered into the levels keyword. If left,\n# it'll default to alphabetical.\ndata <- data.frame(\n time_period=c(\"2018/19\",\"2018/19\",\"2018/19\",\"2018/19\",\"2018/19\",\"2018/19\",\n \"2019/20\",\"2019/20\",\"2019/20\",\"2019/20\",\"2019/20\",\"2019/20\",\n \"2020/21\",\"2020/21\",\"2020/21\",\"2020/21\",\"2020/21\",\"2020/21\"),\n Filter=c(gss_categorical_palette$names,\n gss_categorical_palette$names,\n gss_categorical_palette$names),\n indicator1=sample(4:16,18,replace=TRUE) +\n c(gss_categorical_palette$id,gss_categorical_palette$id,gss_categorical_palette$id)*3\n ) %>%\n mutate(\n time_period=as.factor(time_period),\n Filter=factor(Filter,levels=gss_categorical_palette$names)\n )\n\n# Create a line chart\nggplot(data, aes(x=time_period, y=indicator1, group=Filter, colour=Filter )) +\n geom_line(size=1.2) +\n scale_color_manual(values=gss_categorical_palette$hex) +\n theme_classic() +\n theme(\n legend.position = \"bottom\",\n text = element_text(size = 14, family = \"Arial\"),\n strip.text.x = element_text(size = 20)\n ) +\n ylab(\"Indicator 1\") +\n xlab(\"Time period\")\n\n# Create a stacked bar chart\nggplot(data, aes(x=time_period, y=indicator1, fill=Filter )) +\n geom_bar(stat = \"identity\") +\n scale_fill_manual(values=gss_categorical_palette$hex) +\n theme_classic() +\n theme(\n legend.position = \"bottom\",\n text = element_text(size = 14, family = \"Arial\"),\n strip.text.x = element_text(size = 20)\n ) +\n ylab(\"Indicator 1\") +\n xlab(\"Time period\")\nThe above code should produce something along the lines of the following plots:"
+ "objectID": "statistics-production/user-eng.html#improving-your-own-engagmenent",
+ "href": "statistics-production/user-eng.html#improving-your-own-engagmenent",
+ "title": "User engagement",
+ "section": "Improving your own engagmenent",
+ "text": "Improving your own engagmenent\nDownload this top tips document produced for the GSS, as a starting point for understanding how you can improve your user engagement.\nThrough our work engaging users on EES we have built up a list of interested users who have stated a willingness to be contacted about opportunities to be involved in user testing of our statistics on the platform, if you have any suggestions for things for us to test on EES with these users, please contact explore.statistics@education.gov.uk.\nMore advice and guidance will follow in this space. In the meantime, if you’re interested in user engagement please contact Heather Brown."
},
{
- "objectID": "statistics-production/ees.html#content",
- "href": "statistics-production/ees.html#content",
- "title": "Explore Education Statistics",
- "section": "Content",
- "text": "Content\nIn the content tab you can now start creating your release, embedding the data blocks you’ve created as you go.\n\nYou can use the page view toggles, that float in the bottom left of the page, to jump between edit and preview mode for the release and to view a preview of the table tool.\n\nAdd any headline and/or key stats and figures for your release in the headline facts and figures section.\nYou can then create accordion sections to start adding your main release commentary. These sections are made up of text blocks and data blocks which can be reordered as needed.\n\nHeadline facts and figures\nUse the ‘Add secondary stats’ button to add a data block to your headline stats section.\nUse the ‘Key stats’ options to add key statistic tiles to your release. For each tile you first have to have created a data block that contains only one number.\n\nAfter embedding a key stat tile you can then edit it to add trend information and a description of what the indicator is.\n\n\n\n\n\n\n\nElement\nContent\n\n\n\n\nIndicator name\nAutomatically generated from your data\n\n\nLatest value\nAutomatically generated from your data\n\n\nTrend\nA short one-sentence description of the trend; try to avoid only stating the change from the previous year and talk about the longer-term trend where appropriate\n\n\nGuidance title\nE.g. ‘What is NEET?’ or ‘What are permanent exclusions?’\n\n\nGuidance text\nA simple description in plain English of what the indicator is\n\n\n\n\n\n\nDrafting text\n\n\nAbout these statistics\n\nWe recommend that the first accordion section in each release should be ‘About these statistics’. Do not assume that users will read it, the nature of interactive pages means that the reader may start at any accordion section so remember to include essential information in the section to which it refers.\nAvoid filling this section with:\n\nCrucial caveats or information necessary for accurate interpretation of the statistics, these should be included within the main commentary next to the point they relate to\nTechnical information that is not relevant to the interpretation of the statistics should be saved for the methodology\nDefinitions should be included within the main commentary when they are first mentioned\nLists or descriptions of what is in the commentary as these should be clear from the contents and accordion headings\n\nTry to focus the ‘About these statistics’ section on:\n\nSay why the data are collected and what they are used for or could be used for, including relevant policies and targets\nThe different geographical levels for which data is published\nNon-essential but important information about the statistics, such as:\n\nThe data collection window, mode of data collection and response rates in terms of whether these are representative\nClarification of the population coverage\nChanges in data collection or analysis methods from previous releases\n\nLinks to relevant related statistics including the cause of any differences\nFurther useful detail on specific measures used in the release\nNon-urgent notices to users such as minor error corrections and delays or changes to the next publication. If there is something important which you want all users to know about then give it its own accordion section.\n\n\n\n\n\nFootnotes\n\n\nDo not use footnotes in the text of your content. They’re designed for reference in print, not web pages. If the information in the footnotes is important, include it in the body text. If it’s not, leave it out.\n\n\n\nTables in content\n\nAny data tables should be included as data blocks, however you can also embed static html tables within a text box. These should only be used to present textual tables or for any small presentations of data that are not possible to do in a data block at the moment.\n\n\nYou can create static html tables for presenting information that isn’t embedded in a data block. However, remember that all of the data included or referred to in your content should be available (or createable) from the downloadable open data files.\n\n\n\nFootnotes for tables in content\n\nIf you are including a table in text that needs footnotes, it’s generally advised to include this in the commentary surrounding the table. However, if you think a footnote is still necessary, then we advise writing out the word ‘note’, with the number of the note you need to refer to, and put it in square brackets, for example: ‘Number of people in employment [note 1]’. For more guidance on footnotes outside of EES, see the Analysis Function guidance on symbols, footnotes and codes.\n\n\n\n\nReviewing content\n\nWhile a release is in draft mode, comments can be added to text to help teams collaborate. Simply highlight the text you want to comment on and click on the speech bubble in the editing bar to add a comment.\n\nComments can be edited at a later date, and can also be marked as resolved so that you can see which comments have been addressed and which are still outstanding.\nWhen someone is editing a text box, it will now be instantly frozen for all other users preventing two users from editing the same block of text at the same time. You will be able to see the name of the user who is making edits, and will see the edits coming through every few seconds as their changes autosave."
+ "objectID": "statistics-production/user-eng.html#case-studies",
+ "href": "statistics-production/user-eng.html#case-studies",
+ "title": "User engagement",
+ "section": "Case studies",
+ "text": "Case studies\n\nExplore Education Statistics service\n\nExplore Education Statistics was born from a discovery project, where the emphasis of the discovery was to understand who the users of the Department’s official statistics are, what they need, and where the DfE are currently meeting, exceeding or failing to meet those needs. We spoke to over 90 users directly, via 1 to 1 sessions or workshops and had over 130 survey responses. We used this research to create user personas and user stories which we then assessed ourselves against to see how well we were meeting those needs - the result of this was a recommendation that we needed to do something better. Our discovery report is published on gov.uk.\nFollowing the discovery we moved through to the alpha phase, where we built prototypes and carried out extensive user testing to firm up the functionality that might be required - focussing on different user journeys and the user experience at each step. We then passed a GDS service assessment where our approach was tested against the GDS service standard before moving into the private beta phase where we started building the service for real. We user-tested every bit of the journey thoroughly and used feedback from users to iterate on the functionality we were making available. User testing was primarily through 1 on 1 sessions where users would be set activities to work through and we would watch how the user clicked through each part of the relevant journey and to ask what they might expect to see at any different steps of the process - both when creating/publishing statistics releases and using releases after publishing. We also ran workshops and carried out user surveys to collect as much evidence as possible, for more information on this take a look at the write up of our private beta phase on Hive IT’s website.\n\nIn March 2020 we wrapped up the build of the service and were given the green light via a GDS service assessment to move into our public beta phase. That’s when we started using the service for real and teams started publishing statistics via EES only. Ongoing user feedback has been really important throughout public beta as it has helped us move the service from a minimal viable product to one that works in the best way possible for everyone. We also use this engagement to help us understand how well we are meeting our KPIs, which we review regularly to see how well we are meeting the original aims of the service thinking back to our original discovery and what users told us they needed. There are a number of ongoing feedback routes open to users during the public beta phase of the service:\n\nBeta banner (service focus) Users can submit feedback through the beta banner from any EES page, and responses are managed by the statistics development team. Feedback so far has mostly been aimed at the service functionality more generally but we do regularly get publication specific feedback which we share with the relevant teams.\nFormal user testing (service focus) Our EES team will speak to users via user testing sessions to help inform future iterations to functionality.\nIn-release feedback requests (publication focus) Some teams have been including specific calls for feedback within their release pages e.g. - https://explore-education-statistics.service.gov.uk/find-statistics/apprenticeships-and-traineeships\nGoogle analytics We collect user analytics to help us understand what users are doing with our statistics. This information is shared with production teams via our EES analytics app.\n\nFor more information, get in touch with the team at explore.statistics@education.gov.uk.\n\n\n\nSchool places local authority scorecard\n\nIn 2021 the annual ‘Local Authority School Places Scorecard’ publication was cancelled, due to Covid- 19, allowing time for user-engagement and development of the scorecard. This is a dashboard publication allowing users to select a specific local authority in England, to view it’s progress in providing quality school places. Firstly, internal feedback was sought and received, and generally users found the Scorecard a useful tool. Longer-term additions were identified, however short-term development was also needed to improve the accessibility of the Scorecard so this was the first developmental focus. The Scorecard has been previously published as an excel-based dashboard, however this method of publication was no longer viable due to new rules and regulations for official statistics publications. A new platform was therefore needed and R shiny was suggested as an accessible platform. A new version of a previously published scorecard was produced in R shiny. This was then published externally to gather feedback from users to check they could obtain the required information. The scorecard contained a link to a short survey and over 90 internal and external users responded (after email invitations were sent allowing a month to complete). Most users found the R shiny version easier to use compared to the excel based dashboard which was positive. Suggestions for improvement were gathered and quite a few users mentioned they missed the one-page format of the excel scorecard (the R shiny version has tabs to get to some visualisations) and found some of the newer charts slightly confusing. Therefore solutions were proposed e.g. a downloadable one page PDF summary and alternative charts/clearer guidance. Development work is still ongoing with the new R shiny scorecard due for publication in June 2022. When published this should include improvements to address the most frequent suggestions mentioned in the survey, to ensure users can gather the information they need from the scorecard. For any further information contact Natalie Paterson in the Pupil Place Planning Data team.\n\n\n\nCIN-CLA outcomes statistical release\n\nIn 2020 we created a new statistical release which incorporated changes to the annual Outcomes for children looked after by LAs statistical release and outcomes within the Characteristics of children in need statistical release, proposed by the department in September 2020. Improving these statistics was a commitment made at the end of the CIN review. As part of the review of the National Statistics, an ad-hoc release was published (Outcomes of children in need including looked after children) and we asked users to submit feedback on the proposals through the running of a consultation via Gov.UK to which people could respond (Children in need and looked after children statistics: proposed changes). We involved policy colleagues (including policy analysts) in the proposed changes ahead of publishing. We engaged with colleagues across the department to identify known users and user groups and notified them of the request for feedback on the proposals We also followed up with a reminder as we got closer to the deadline. As part of this process, we had a meeting with the main stakeholders (the board of the National Association of Virtual School Heads). These key were very keen to respond to the proposals but were having difficulty with resource and time to provide a comprehensive response, so we agreed to have a meeting to talk through everything and we took notes. Reaching out to users and actively engaging with them produced a great result - we received a wide range of feedback from a range of sources including previously unknown users. We received external and internal feedback. We published the results of the feedback on the proposals on gov.uk, including a summary of the feedback received and what we were able to incorporate into the new publication. This was not an official Government consultation and we had to negotiate with gov.uk on how to publish the documents. We were advised that going forward, it would be preferable for user engagement of this type to be published alongside the statistical releases as part of the new EES platform. For any further information contact Bree Waine."
},
{
- "objectID": "statistics-production/ees.html#embedding-r-shiny-blocks",
- "href": "statistics-production/ees.html#embedding-r-shiny-blocks",
- "title": "Explore Education Statistics",
- "section": "Embedding R-Shiny blocks",
- "text": "Embedding R-Shiny blocks\nIf you need to include a type of chart that isn’t possible using the standard EES chart options, for example an interactive chart with filters, you can embed a block to display a custom R-Shiny produced chart.\nTo embed a shiny chart, you’ll first need to create an R-Shiny application containing the chart using the DfE tiny-shiny template and get it hosted on our DfE GitHub and ShinyApps accounts. More details on both of these are available in the tiny-shiny section of this guidance.\nOnce you’ve got the R-Shiny app set up and hosted, you can embed it using the Add embed block button in the content area. This will give you the option to enter a URL, where you can enter the URL of the shiny app. Valid URLs to use in the embed block dialogue box are limited to only those on the DfE ShinyApps server (and the internal rsconnect when needed for draft publications).\n\n\n\nThe EES interface for embedding a Shiny chart"
+ "objectID": "statistics-production/user-eng.html#what-is-user-engagement",
+ "href": "statistics-production/user-eng.html#what-is-user-engagement",
+ "title": "User engagement",
+ "section": "What is user engagement?",
+ "text": "What is user engagement?\n\n\n\n\nThe ONS have also published ten top tips to help improve user engagement."
},
{
- "objectID": "statistics-production/ees.html#glossary",
- "href": "statistics-production/ees.html#glossary",
- "title": "Explore Education Statistics",
- "section": "Glossary",
- "text": "Glossary\nOur glossary on EES is a growing page that helps us to standardise how refer to key terms and phrases across all of Official statistics - https://explore-education-statistics.service.gov.uk/glossary.\n\n\nAdding new definitions\n\nContact us at explore.statistics@education.gov.uk, with the title and definition and we can add this for you. It’s worth having a check on the glossary for similar or related terms, and whether you should be looking to harmonise with other teams.\n\n\n\nLinking to definitions\n\nYou can link to any term in the glossary by appending the glossary url with #name-of-definition, replacing any spaces with hyphens.\nFor example, to link to the definition for ‘Respite care’, you would use the following link:\nhttps://explore-education-statistics.service.gov.uk/glossary#respite-care\nYou can test this works by typing the url into your browser, it should take you to that specific definition on the glossary page. If you’re unsure at all, or have special characters in the title of your glossary entry, please ask us for support on getting the right link.\n\n\n\nLinking from your release\n\nWhen writing your release content, highlight a word or phrase and click the link icon in the text editor bar.\n\nThen, paste your glossary url into the box that appears.\n\nThat’s it, the system will automatically recognise that the link is for the glossary and will do the rest. You can then change to the preview mode to see how this would then appear to public users, and test that the box appears with the definition."
+ "objectID": "statistics-production/user-eng.html#user-engagement-successes",
+ "href": "statistics-production/user-eng.html#user-engagement-successes",
+ "title": "User engagement",
+ "section": "User engagement successes",
+ "text": "User engagement successes"
},
{
- "objectID": "statistics-production/ees.html#sign-off",
- "href": "statistics-production/ees.html#sign-off",
- "title": "Explore Education Statistics",
- "section": "Sign off",
- "text": "Sign off\nOnce you’re happy with your release you need to go to the sign off page and change it’s status in order to move it through the release pipeline.\nThere are three statuses:\n\nIn draft (where the production team work on drafting the release)\nReady for higher review (where the senior statistician checks over the release before approving)\nApproved for publication (after approval has been given, releases in this status will be published on their scheduled date)\n\n\nOnly users with approver permissions (usually G6 or above) can sign off the release for publication.\n\n\n\nErrors and warnings\n\nThere’s quite a few things to remember to do as you build your release so to help ensure you haven’t missed anything a release checklist is also available via the sign off page\nThe checklist can be accessed by clicking to edit release status.\nRemember to check over it before you submit your release for approval as a release that has outstanding warnings on it will not be able to be published.\n\n\n\n\nHow to approve a release\n\nReleases are approved via the sign off tab. The release date is also set during this stage of the process. All releases scheduled for a specific date will be published at 9:30am on that day.\n\nThe approver has the ability to approve the release to be published on a specific date or to publish as soon as possible. Publishing as soon as possible is useful for publishing amendments to existing releases.\n\nThis page also gives you the expected release url which may be useful to know for other things, for example, sending to the web team to add to your gov.uk announcement page.\n\n\n\n\nNext release expected\n\nYou also have the option to add a date for when the next release is expected. This will appear at the top of a release page and give users an idea of when to expect the next release in the publication series. You can provide the planned month and year of the next release within this publication.\nIf this is added but then needs to be changed or removed later in the year it is easy to do so, by creating an amendment and setting a new next release expected date on sign-off."
+ "objectID": "statistics-production/user-eng.html#consultations",
+ "href": "statistics-production/user-eng.html#consultations",
+ "title": "User engagement",
+ "section": "Consultations",
+ "text": "Consultations\nIf you are considering changes to your publication and are contemplating a formal consultation do get in touch with HOP office early on to discuss.\nTo read more about consultations for background information please see the following two links:\n\nCode of Practice on Consultations\nConsultation principles: guidance - WWW.GOV.UK\n\nWith the first of the above links, please bear in mind this was written back in 2008 so certain aspects may have been updated but the background and principles do remain the same.\nSome examples of previous consultations are included below:\n\nOutcome and experience data - Office for Students\nConsultation on Data Futures and data collection - Office for Students\nSurveys on childcare and early years in England - GOV.UK (www.gov.uk)\nProposed cessation of ‘Income related benefits: estimates of take-up’ statistics - GOV.UK (www.gov.uk)\nProposals for a new statistical series to count unemployed claimants - GOV.UK (www.gov.uk)"
},
{
- "objectID": "statistics-production/ees.html#pre-release-access",
- "href": "statistics-production/ees.html#pre-release-access",
- "title": "Explore Education Statistics",
- "section": "Pre-release access",
- "text": "Pre-release access\nPrior to each release going live the production team are also able to grant pre-release access to a named group of users 24 hours before it goes live. These users do not require full access to the whole admin service. They will be able to see preview versions of any releases they have been granted access to.\nThis preview is only accessible for the 24 hours before the publication date, the emails may go out to users before then. We expect teams will continue to send an email at 09:30 on pre-release day, including any additional briefing and the link to the pre-release area.\n\nJobshare emails should not be sent invites for pre-release. The personal emails for the specific individuals should be used instead, as jobshare emails do not have active accounts with DfE to access EES.\n\n\nPre-release access via Explore Education Statistics is currently limited to DFE users only, if you need to share your release with external users you will need to do so outside of the system.\n\n\n\n\n\nEES PRA one pager\n\n\n\n\n\n\n\n\n\n\n\nInviting users for pre-release access and building the public pre-release list can be found within the ‘Pre-release access’ tab on the dashboard.\nOnce the release has been marked as approved, go to the ‘Pre-release access’ tab and add the relevant email addresses to grant pre-release access. All invited users will receive an email to say that they have been given pre-release access and will get a url where the preview release will be available.\n\nYou should also create the public facing pre-release access page by clicking the ‘Public access list’ tab.\n\nAfter creating your pre-release access list a text editor will appear where you can list the roles that have been given early access to the release. This list will then appear in the list of file downloads at the top of each release."
+ "objectID": "statistics-production/user-eng.html#surveys",
+ "href": "statistics-production/user-eng.html#surveys",
+ "title": "User engagement",
+ "section": "Surveys",
+ "text": "Surveys\nWe routinely publish statistical releases, but who is using these and why? It’s important to understand your user types and need, so we (public and department) get best value from what we do. Using these insights can help us to continually improve our products and services. Having a survey with your releases is a great way to glean this feedback. Examples of statistical publication surveys, as well as ONS top tips on user surveys are below/attached. It would be great to include your examples too, as well as capture your lessons learned and tips, so do send these to us. If you do want further advice, please contact Heather Brown or Ian Hewson.\nUser survey and questionnaire design top tips (word document)\nResults from survey of statistics users (word document)\n\n\nExamples\n\nHigher Education Institutions Enrolments and Qualifications statistical bulletins - readership survey September 2019 (economy-ni.gov.uk)\nFurther Education & skills/Apprenticeships and traineeships release feedback survey - IH (Edit) Microsoft Forms (office.com)\nHigher Education Statistics - Department for Education - Citizen Space\nStatistics-and-data_user-feedback-survey_Land-and-Property-Data-Team_RoS.._.pdf"
},
{
- "objectID": "statistics-production/ees.html#release-notes",
- "href": "statistics-production/ees.html#release-notes",
- "title": "Explore Education Statistics",
- "section": "Release notes",
- "text": "Release notes\nWhen publishing a new amendment you should add a ‘release note’ to your release so users can be informed of what has changed:"
+ "objectID": "statistics-production/pub.html",
+ "href": "statistics-production/pub.html",
+ "title": "Publication checklists",
+ "section": "",
+ "text": "Guidance for how to publish different types of statistics"
},
{
- "objectID": "statistics-production/ees.html#methodology-formatting",
- "href": "statistics-production/ees.html#methodology-formatting",
- "title": "Explore Education Statistics",
- "section": "Methodology formatting",
- "text": "Methodology formatting\nSubheadings make it easier for your users to navigate through your methodology. When editing text blocks in your methodology, you can do this by highlighting your subheading and selecting “Heading 3” in place of “Paragraph”:\n\nYou can add further subheadings underneath this but try to limit the number of subheadings per accordion to 10 at the most. Too many subheadings will make it tricky for users to find what they are looking for."
+ "objectID": "statistics-production/pub.html#how-to-publish",
+ "href": "statistics-production/pub.html#how-to-publish",
+ "title": "Publication checklists",
+ "section": "How to publish",
+ "text": "How to publish\n\n\n\nEES create release one pager"
},
{
- "objectID": "statistics-production/ees.html#methodology-images",
- "href": "statistics-production/ees.html#methodology-images",
- "title": "Explore Education Statistics",
- "section": "Methodology images",
- "text": "Methodology images\nYou can add images to methodology pages via the content editor.\n\nWhen uploading an image you will need to add alt text via by clicking the eye symbol, you may also choose to add an image caption.\n\n\nAccessibility matters for uploaded images too, consider the following if you do need to upload an image -\n\nKeep them simple\nUse GDS colours\nDon’t put borders around images\nEnsure there is a high enough colour contrast ratio between elements. You can use a colour contrast checker to check the colours you’re using\nTry to avoid adding text to your images other than axis labels and limited annotations."
+ "objectID": "statistics-production/pub.html#publication-checklist",
+ "href": "statistics-production/pub.html#publication-checklist",
+ "title": "Publication checklists",
+ "section": "Publication checklist",
+ "text": "Publication checklist\nBefore releasing statistics for the first time you may want to discuss the new process with key stakeholders and/or pre-release users to make them aware of the new service. You should also inform the Explore Statistics Mailbox and HoP Office teams.\nBefore you start creating a release in the platform you should have:\n\nAnnounced the upcoming release via gov.uk\nSent metadata form to HoP\nContacted the BAU team so we can support you with your first release\nProduced your tidy csv data files with appropriate disclosure control\nProduced metadata files for each csv data file\nRan your data and metadata through our screener checks\n\nBefore you publish a release you have created in the platform you should have:\n\nChecked all the data has loaded successfully\nWritten footnotes\nWritten content (including tables and charts)\nCreated a data guidance document\nCreated a public pre-release access list\nEnsured methodology information is either linked off to or attached to the release\nPassed the release for higher review (senior sign-off)\nPreviewed your release\nScheduled the release date\nInvited your PRA list to preview 24 hours before it goes live\nRaised a web ticket for the associated gov.uk page\n\n\nWord templates for the data guidance, pra-list, and content can be found on sharepoint."
+ },
+ {
+ "objectID": "statistics-production/pub.html#linking-to-gov.uk",
+ "href": "statistics-production/pub.html#linking-to-gov.uk",
+ "title": "Publication checklists",
+ "section": "Linking to gov.uk",
+ "text": "Linking to gov.uk\nYou will need to arrange a gov.uk statistics publication page so that it links to EES. Here is how to do that:\nTwo days ahead of publication, you’ll need to raise a ticket with the Digital communications (gov.uk) team and ask them to create a new gov.uk statistics page with a link to EES, connect it to the announcement and add to any collections.\nIn your request, you’ll need to include:\n\ntitle, summary sentence and ‘detail’ for the new page – you can include a link to previous releases if you want it to be the same\nthe link for your EES release – if you don’t have it you can update the ticket when the link is available\nthe link for the announcement\nthe link of any gov.uk collections it needs to be added to\nan email with clearance from your deputy director\nan email confirming communications are happy for it to go if it’s for stats that aren’t pre-announced or it’s an update to stats made after publication – so that they can prepare reactive lines\n\nHere are examples of how the page will look like:\n\nPrimary School Performance Tables 2018\nHigher Education Student Statistics 2018 to 2019\n\n\nYou can find what the link to your EES release will be by looking at the ‘Sign off’ page within the release dashboard on EES.\n\n\nText to update the collection page:\nIn addition, you will also need to raise a ticket with the Digital communications (gov.uk) team to update your collection page/s with the following text:\n\nFrom [publication date], the Department for Education will be publishing the [publication title] on a new statistics site, Explore Education Statistics (EES).\n\nWhen you raise the ticket please ensure that all the documents you submit are in an accessible format as stated in the accessibility guidance. There is also guidance on accessibility for Excel workbooks in Teams."
},
{
- "objectID": "statistics-production/ees.html#publication-management",
- "href": "statistics-production/ees.html#publication-management",
- "title": "Explore Education Statistics",
- "section": "Publication management",
- "text": "Publication management\nUse this page to change the details for your publication, such as the contact information or to add links to legacy releases on gov.uk.\nAny updates here will affect all releases in the series and will be published immediately.\n\nIt is possible to change the publication title via this page, however this should be used rarely and will not affect the publications url if one or more published releases exist."
+ "objectID": "statistics-production/pub.html#how-to-publish-1",
+ "href": "statistics-production/pub.html#how-to-publish-1",
+ "title": "Publication checklists",
+ "section": "How to publish",
+ "text": "How to publish\nAhead of publication, you’ll need to raise a ticket with the Digital communications (gov.uk) team and ask them to create a new release (or add to a series if this already exists).\nIn your request, you’ll need to include:\n\ntitle, summary sentence and ‘detail’ for the new page\nthe link for the announcement\nthe link of any gov.uk collections it needs to be added to\nan attachment with the accessible data files\nan email with clearance from your deputy director\n\nWhen you raise the ticket please ensure that all the documents you submit are in an accessible format as stated in the accessibility guidance. There is also guidance on accessibility for Excel workbooks in Teams."
},
{
- "objectID": "statistics-production/ees.html#create-new-release",
- "href": "statistics-production/ees.html#create-new-release",
- "title": "Explore Education Statistics",
- "section": "Create new release",
- "text": "Create new release\nAfter finding the publication you want to create a release for, just press the button to create a new release.\nCurrently the following types of release can be created in EES:\n\nNational Statistics (includes OSR tick mark logo at the top of the release page)\nOfficial Statistics\nAdhoc Statistics\n\nWhen creating a release you will be asked to fill in some release summary fields.\n\nThe time period for the release should reflect the time period of the data that this latest release adds to the time series."
+ "objectID": "statistics-production/pub.html#publication-checklist-1",
+ "href": "statistics-production/pub.html#publication-checklist-1",
+ "title": "Publication checklists",
+ "section": "Publication checklist",
+ "text": "Publication checklist\nBefore you start creating your release you should have:\n\nAnnounced the upcoming release via gov.uk\nSent metadata form to HoP\n\nBefore you publish a release you should have:\n\nChecked that the data meets accessibility requirements, ideally as a .csv file, or an accessible Excel file where this is not possible\nWritten footnotes where appropriate\nPassed the release for higher review (senior sign-off)\nSent the final files and sign-off email to the digital communications (gov.uk) team"
},
{
- "objectID": "statistics-production/ees.html#managing-legacy-releases",
- "href": "statistics-production/ees.html#managing-legacy-releases",
- "title": "Explore Education Statistics",
- "section": "Managing legacy releases",
- "text": "Managing legacy releases\nPast publications on other services can be added to the previous releases links that appear on the top right of a release page by using the ‘Manage legacy releases’ section. In here you can add links to previous releases and choose the order in which they appear.\nThis section can be found by publication owners in the publication management section."
+ "objectID": "statistics-production/pub.html#how-to-publish-2",
+ "href": "statistics-production/pub.html#how-to-publish-2",
+ "title": "Publication checklists",
+ "section": "How to publish",
+ "text": "How to publish\nWe recommend that teams use R Shiny to build and publish dashboards. Shiny applications are easily made accessible, and underlying code for Shiny apps can be published alongside for transparency.\nFor more information on publishing dashboards, including template code and how to get started, see our Public dashboards page."
},
{
- "objectID": "statistics-production/examples.html",
- "href": "statistics-production/examples.html",
- "title": "Good examples in EES",
+ "objectID": "statistics-production/embedded-charts.html",
+ "href": "statistics-production/embedded-charts.html",
+ "title": "Embedded visualisations in EES",
"section": "",
- "text": "Examples of good practice in EES, from file names to content"
+ "text": "Guidance for creating and embedding R-Shiny visualisations in EES publications"
},
{
- "objectID": "statistics-production/examples.html#files",
- "href": "statistics-production/examples.html#files",
- "title": "Good examples in EES",
- "section": "Files",
- "text": "Files\nFiles can be downloaded from multiple areas across the platform and often users will judge their contents based on the name alone. For display names, it’s important we make it as clear as possible to users as to what each file contains without cluttering the title with information that is already available from the page around it, saving detailed coverage for the data guidance.\n\n\nFile names\n\nFull guidance for naming files.\nThe Education and training statistics for the UK release has some good examples of file names which clearly explain what is in the file, while falling well under 35-50 characters in length, e.g.:\n\nuk_schools.csv\nuk_expenditure.csv\nuk_pupils.csv\n\nThe file names do not have time periods in them, making them easier for users to make use of newer versions in future\n\n\n\nSubject names\n\nFull guidance for naming subject files.\nThe School Workforce Census applies this guidance well, with short and simple titles that make sense to a non-expert, e.g.:\n\nTeacher pay\nTeacher retention\nSubjects taught\n\nThese titles are not cluttered with geographies and time periods, which automatically populate in the subject metadata.\n\n\n\nSupporting files\n\nFull guidance for using supporting files in EES.\nThe Initial Teacher Training performance profiles publication is one example of needing to upload a file outside of the regular data/metadata structure. Their provider-level tables also contain national and regional data, which is not supported in EES at the time of writing. The file has a clear title, and data guidance clearly explains what is in the file."
+ "objectID": "statistics-production/embedded-charts.html#when-to-use-an-embedded-chart",
+ "href": "statistics-production/embedded-charts.html#when-to-use-an-embedded-chart",
+ "title": "Embedded visualisations in EES",
+ "section": "When to use an embedded chart",
+ "text": "When to use an embedded chart\nEES provides a wide range of inbuilt chart options and will always be the first preference for static line, bar and geographical charts. This helps us to clearly maintain consistent styling and accessibility levels across the site.\nHowever, there are some instances where you might want to publish something that we can’t provide through EES. Example use cases would be:\n\ninteractive charts controlled by drop-down filters;\nchart types not provided by EES, e.g. sankey diagrams, box plots, waffle charts, pie charts, dumbbell charts, etc."
},
{
- "objectID": "statistics-production/examples.html#data-guidance",
- "href": "statistics-production/examples.html#data-guidance",
- "title": "Good examples in EES",
- "section": "Data guidance",
- "text": "Data guidance\nThe public data guidance is a key element of your release, to help direct users to the right file to download or build tables with. It is frequently highlighted by users of our stats as a really helpful element of EES. You will be prompted to fill in data guidance in the release checklist. Any release that does not have a complete public data guidance will not be blocked from approval and publishing.\n\n\nPublic data guidance\n\nFull guidance for public data guidance.\nThe School Workforce Census, along with having sensible brief file names, also has great examples of useful data guidance, which explain what is in each file without overloading the user with information:\n\n\n\n\nPublic data guidance - overview\n\nFull guidance for public data guidance.\nThe NEET annual brief is a good example of a public metadata overview section done well. It follows the EES template, goes into just the right amount of detail, and is easy to understand.\n\nNEET data guidance on EES."
+ "objectID": "statistics-production/embedded-charts.html#tools",
+ "href": "statistics-production/embedded-charts.html#tools",
+ "title": "Embedded visualisations in EES",
+ "section": "Tools",
+ "text": "Tools\nWe currently only support custom charts created using R-Shiny. These should be created with ggplot and plotly. We provide a template example of a demo R-Shiny/ggplot chart on the DfE Analytical Services GitHub site, which is described below."
},
{
- "objectID": "statistics-production/examples.html#charts-and-tables",
- "href": "statistics-production/examples.html#charts-and-tables",
- "title": "Good examples in EES",
- "section": "Charts and tables",
- "text": "Charts and tables\nCharts and tables are key ways to highlight important information and stories in your release. They should be clear, contain few footnotes that are essential to interpreting the chart or table, and have good descriptive alternative text so users with screen readers can understand what is being visualised.\n\n\nOther infographics\n\nFull guidance on using charts built outside of EES.\nThe Further Education: outcome based success measures release contains infographics that cannot be built in EES. They are saved as SVGs so render clearly on any device, and follows the same colour palette as the rest of the release.\n\n\n\n\nCharts and footnotes\n\nFull guidance on creating charts, and guidance on creating footnotes.\nThe Widening participation in higher education publication has a good example of a chart built with a long timeseries in the chart builder, with sparing use of footnotes. For example below, a line in the chart indicates a point where comparisons cannot be made, and a key caveat around the comparability of years is included as a footnote, which is needed to interpret the data and chart accurately.\n\n\n\n\nAccessibility and alt-text\n\nGood alt-text descriptions in charts will not just repeat the title of the chart, but instead describe the type of chart, what the data coverage is, and what trends can be seen in the chart to a user accessing the page with a screenreader.\nThe school workforce census has a good example of alt-text for one of their charts. The associated alt-text for the chart below is: “Line chart showing the percentage of all teachers taking absence across all state funded schools in England between the academic years 2014/15 and 2018/19. The chart shows a fall from 55.7% to 54.0% in this period.”\n\n\n\n\nFeatured tables\n\nFull guidance on creating featured tables.\nGood featured tables have short, easy to understand titles, and point users to commonly requested tables or other tables that might be interesting to a wide range of users. For example, the pupil absence in schools: autumn term release has a good example of a clear title, with further information (including date and geographic coverage) left in the description. Leaving these out of the main title makes it much easier for users to navigate through multiple featured tables:"
+ "objectID": "statistics-production/embedded-charts.html#review-and-authorisation",
+ "href": "statistics-production/embedded-charts.html#review-and-authorisation",
+ "title": "Embedded visualisations in EES",
+ "section": "Review and authorisation",
+ "text": "Review and authorisation\nTo get a custom chart approved for embedding within a publication, you’ll need to get it reviewed by the Statistics Development team (in addition to your standard approval chain)."
},
{
- "objectID": "statistics-production/examples.html#content",
- "href": "statistics-production/examples.html#content",
- "title": "Good examples in EES",
- "section": "Content",
- "text": "Content\nClear, concise content is required to direct your users to the right place, and keep them engaged with your release. Our content guidance page contains a raft of great advice on how to structure your release and write for the general public.\n\n\nHeadlines\n\nFull guidance on creating a headline section\nThe Education, health and care plans release has a good example of a headlines section. There are a sensible number of key statistics, and the summary below gives an overview of trends without diving into the numbers and overwhelming the user.\n\nThe Exclusions release has a good example of custom explanations under key stat tiles to show users exactly what the numbers are describing:\n\n\n\n\nAccordion content\n\nFull guidance on writing content for accordions\nThe parental responsibility measures release has good examples of writing for the public in accordion content. Content in each accordion is short and follows the “pyramid” principle of having essential information at the top, summarising the trend, then going into detail at the bottom.\n\n\n\nActive subheadings\n\nFull guidance on using active headings and titles\nThe EHCP release has some great examples of active subheadings within accordions, which explain the overall trends without overloading users with detail.\n\n\n\nGlossary links\n\nThe summary of the CIN / CLA outcomes release has examples of this in action.\n\n\n\nMethodology\n\nFull guidance on creating a methodology in EES.\nGood methodologies will broadly cover topics in our recommended methodology template. The Graduate outcomes (LEO) methodology broadly follows this outline, making use of effective formatting in EES to help users navigate through accordions, although definitions can be moved to the EES glossary.\nThe Key stage 4 destinations methodology makes good use of annexes to show changes to their methodology over time."
+ "objectID": "statistics-production/embedded-charts.html#the-dfe-tiny-shiny-template",
+ "href": "statistics-production/embedded-charts.html#the-dfe-tiny-shiny-template",
+ "title": "Embedded visualisations in EES",
+ "section": "The DfE Tiny-Shiny template",
+ "text": "The DfE Tiny-Shiny template\nOur template tiny shiny app repository should be used a starting point for all embedded shiny charts.\nTo get an app set-up for use with EES, you’ll need the Statistics Development Team team to create a repo for the app within the DfE Analytical Services area on GitHub."
},
{
- "objectID": "statistics-production/examples.html#approvals-and-amendments",
- "href": "statistics-production/examples.html#approvals-and-amendments",
- "title": "Good examples in EES",
- "section": "Approvals and amendments",
- "text": "Approvals and amendments\nKeeping track of processes for approving and amending releases is crucial for transparency. Internally, there should be clear paper trail of who has signed off the release, and externally, users need to know if anything in the release has changed from the last time they saw it.\n\n\nPublic amendment notes\n\nFull guidance for creating public amendment notes.\nThe Pupil absence in England - Autumn and Spring terms publication has some good examples of release notes.\nThese notes clearly explain to the user what has changed, which files were affected and at what level:\n\n\n\n\nInternal release status notes\n\nFull guidance for creating public amendment notes.\nThe COVID attendance publication has fortnightly releases at the time of writing, and the team have a good process for sign-off. This includes detailed information at each stage to confirm who has signed off, who has requested an action be taken, and who has carried out the action."
+ "objectID": "statistics-production/embedded-charts.html#whats-in-the-template",
+ "href": "statistics-production/embedded-charts.html#whats-in-the-template",
+ "title": "Embedded visualisations in EES",
+ "section": "What’s in the template",
+ "text": "What’s in the template\nThe template provides code for some basic interactive plots. Each example plot is contained with one of the existing branches below for demonstration purposes:\n\ndemo-interactive-bar\ndemo-interactive-line"
},
{
- "objectID": "statistics-production/stats_tools.html",
- "href": "statistics-production/stats_tools.html",
- "title": "Tools for statistics",
- "section": "",
- "text": "Guidance for how to use the tools available for statistics publishers\n\n\nWe also have a number of tools for statistics production teams to use to help them in their processes, they can be accessed at the links below when using a DfE device.\n\nEES data screening app\nThis application allows you to screen your data files against the underlying data standards for Official and National statistical publications. If your file passes the checks it also gives you some basic options for exploring the data in the file.\n\nEES data screener\nData screener code on GitHub\n\n\n\nEES analytics\nThere is an R Shiny dashboard that provides access to the Google Analytics data for the Explore Education Statistics platform.\n\nExplore Education Statistics analytics dashboard\nCode is not yet publicly available, it is stored in Azure DevOps, contact us if you are interested in seeing it or contributing.\n\n\n\nPublication RAP self-assessment tool\nA self assessment tool allowing official statistics publications to understand how their production processes match up against RAP expectations.\n\nRAP self-assessment tool\nRAP self-assessment code mirrored on GitHub\n\n\n\nTemplate code\n\nTemplate DfE R Shiny dashboard\nTemplate QA code repository\ndfeR R package\n\n\n\n\n\n Back to top"
+ "objectID": "statistics-production/embedded-charts.html#working-with-data",
+ "href": "statistics-production/embedded-charts.html#working-with-data",
+ "title": "Embedded visualisations in EES",
+ "section": "Working with data",
+ "text": "Working with data\nAs with the full dashboards, the embedded charts currently require the underlying data to be either included within the app repository on GitHub or uploaded elsewhere publicly accessible such as Google Drive or Dropbox. This currently means that any embedded charts being developed will need to use either dummy data or previously published data until the moment of publication. At the point of the parent release going live, the chart can then be updated with the latest data. Do not upload unpublished data to GitHub, Google Drive or Dropbox.\nAs described earlier, where you need to use unpublished data in your chart prior to publication, you can either a) run the chart locally in R-Studio (without pushing the unpublished data to GitHub) or b) create a DevOps/rsconnect deploy of your app, which can be temporarily used as the embed block URL. Note, this will need updating to a URL to the public dashboard on ShinyApps ready for publication.\nWe are currently developing a route to allow charts via R-Shiny apps to be hosted on DfE servers, such that draft publications will be able to incorporate embedded charts with the unpublished data. The data itself will then be accessed either from a SQL database on DfE servers."
},
{
- "objectID": "statistics-production/user-analytics.html",
- "href": "statistics-production/user-analytics.html",
- "title": "EES analytics",
- "section": "",
- "text": "Guidance on understanding how users are interacting with your statistics published via EES\n\n\nWe track analytics of users on Explore Education Statistics using Google Analytics as well as other tools. These analytics can be found on our EES Analytics dashboard. This tells you a range of things like:\n\nHow many times each accordion has been clicked\nHow long users spend on your page\nHow many times each table has been used in the table builder or downloaded\n\n\n\nUser-led content improvements\n\nThe below table gives some pointers on how you can use these stats to improve your release for users:\n\n\n\n\n\n\n\nMeasure\nGuidance\n\n\n\n\nSearch terms\nFor popular search terms, add additional commentary or signposting using subheadings and accordions for to support your user’s interests. If you refer to a popular search term by another name, consider changing it so users know what you’re talking about.\n\n\nAccordion openings\nConsider removing accordions with low numbers of clicks from your release. You could move them further up the page or group them into existing accordions if this commentary is essential to your release.\n\n\nDownloads\nFor files with low numbers of downloads, consider combining information into other files, or consider removing from your publication entirely if users are no longer interested in this data.\n\n\nTables built\nFor tables with low numbers of interactions in the table tool, create featured tables to demonstrate the useful information they hold.\n\n\nPermalinks\nSave popular permalinks as featured tables in future, so users can easily get to the data they want.\n\n\nTraffic sources\nConsider where your users are being routed from and what is being written about your stats - clearly highlight key stories in the Headlines section and make use of the blue key stat tiles to make them easy to digest.\n\n\n\n\n\n\nGuidance for analytics\n\nPerformance analysis: GDS starter guide to using Google Analytics\nPerformance analysis: GDS Advanced Analytics Reporting Techniques\n\n\n\n\n\n Back to top"
+ "objectID": "statistics-production/embedded-charts.html#specific-design-recommendations-for-embedded-plots",
+ "href": "statistics-production/embedded-charts.html#specific-design-recommendations-for-embedded-plots",
+ "title": "Embedded visualisations in EES",
+ "section": "Specific design recommendations for embedded plots",
+ "text": "Specific design recommendations for embedded plots\nExample code for producing an embeddable shiny chart is given in the template tiny shiny app repository. The following recommendations should be followed in adapting this code:\n\nFigures should be produced using plotly/ggplot2\nFigure lengths and heights should be in the range 6-10cm\nText sizes in plots should be no smaller than 12pt\nPlotly overlays should be turned off\n\nExample code for creating charts using ggplot can be found in the Using Explore Education Statistics guidance"
},
{
- "objectID": "writing-visualising/cd.html",
- "href": "writing-visualising/cd.html",
- "title": "Writing and visualising",
+ "objectID": "CODE_OF_CONDUCT.html",
+ "href": "CODE_OF_CONDUCT.html",
+ "title": "Contributor Covenant Code of Conduct",
"section": "",
- "text": "Things to consider when writing statistical commentary"
+ "text": "We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.\nWe pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.\n\n\n\nExamples of behavior that contributes to a positive environment for our community include:\n\nDemonstrating empathy and kindness toward other people\nBeing respectful of differing opinions, viewpoints, and experiences\nGiving and gracefully accepting constructive feedback\nAccepting responsibility and apologizing to those affected by our mistakes, and learning from the experience\nFocusing on what is best not just for us as individuals, but for the overall community\n\nExamples of unacceptable behavior include:\n\nThe use of sexualized language or imagery, and sexual attention or advances of any kind\nTrolling, insulting or derogatory comments, and personal or political attacks\nPublic or private harassment\nPublishing others’ private information, such as a physical or email address, without their explicit permission\nOther conduct which could reasonably be considered inappropriate in a professional setting\n\n\n\n\nCommunity leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.\nCommunity leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.\n\n\n\nThis Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.\n\n\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at statistics.development@education.gov.uk. All complaints will be reviewed and investigated promptly and fairly.\nAll community leaders are obligated to respect the privacy and security of the reporter of any incident.\n\n\n\nCommunity leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:\n\n\nCommunity Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.\nConsequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.\n\n\n\nCommunity Impact: A violation through a single incident or series of actions.\nConsequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.\n\n\n\nCommunity Impact: A serious violation of community standards, including sustained inappropriate behavior.\nConsequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.\n\n\n\nCommunity Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.\nConsequence: A permanent ban from any sort of public interaction within the community.\n\n\n\n\nThis Code of Conduct is adapted from the Contributor Covenant, version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.\nCommunity Impact Guidelines were inspired by Mozilla’s code of conduct enforcement ladder.\nFor answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations."
},
{
- "objectID": "writing-visualising/cd.html#headline-sections",
- "href": "writing-visualising/cd.html#headline-sections",
- "title": "Writing and visualising",
- "section": "Headline sections",
- "text": "Headline sections\nIt is common to use bullet points to draw out key headline messages, either for policy lines or for general interest. Here are some top tips for writing headlines:\n\nAsk yourself if this is the most important, useful and relevant point to make? Why? What is new?\nWe recommend a maximum of six top headlines\nYou shouldn’t be trying to summarise all the findings in the publication\nFor regular publications, headlines won’t necessarily be the same every time\nHeadlines should be a single sentence making a single point, and be able to stand alone from the publication\nHeadlines should make sense to everyone and anyone (no jargon)\nStructure headlines as: what has happened; why is this important - don’t give numbers without context\nAt least of your headlines should put the latest figures in the context of the longer-term change\nRound figures in headlines, you don’t need lots of decimal places\nIf there is essential context for the headline facts then put this here"
+ "objectID": "CODE_OF_CONDUCT.html#our-pledge",
+ "href": "CODE_OF_CONDUCT.html#our-pledge",
+ "title": "Contributor Covenant Code of Conduct",
+ "section": "",
+ "text": "We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.\nWe pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community."
},
{
- "objectID": "writing-visualising/cd.html#content-headings",
- "href": "writing-visualising/cd.html#content-headings",
- "title": "Writing and visualising",
- "section": "Content headings",
- "text": "Content headings\nThroughout the commentary, active sub-headings outline the main messages making them memorable for users. They should be a short summary of the contents - aim for 10 words or less - rather than a full statistical headline. Avoid using exaggerated or sensationalist language that you would not use in your main commentary such as “highest ever” or “only”. Be neutral in selecting the message to use in the heading rather than favouring positive or negative messages."
+ "objectID": "CODE_OF_CONDUCT.html#our-standards",
+ "href": "CODE_OF_CONDUCT.html#our-standards",
+ "title": "Contributor Covenant Code of Conduct",
+ "section": "",
+ "text": "Examples of behavior that contributes to a positive environment for our community include:\n\nDemonstrating empathy and kindness toward other people\nBeing respectful of differing opinions, viewpoints, and experiences\nGiving and gracefully accepting constructive feedback\nAccepting responsibility and apologizing to those affected by our mistakes, and learning from the experience\nFocusing on what is best not just for us as individuals, but for the overall community\n\nExamples of unacceptable behavior include:\n\nThe use of sexualized language or imagery, and sexual attention or advances of any kind\nTrolling, insulting or derogatory comments, and personal or political attacks\nPublic or private harassment\nPublishing others’ private information, such as a physical or email address, without their explicit permission\nOther conduct which could reasonably be considered inappropriate in a professional setting"
},
{
- "objectID": "writing-visualising/cd.html#accordion-section-content",
- "href": "writing-visualising/cd.html#accordion-section-content",
- "title": "Writing and visualising",
- "section": "Accordion section content",
- "text": "Accordion section content\nYou should split your release into sections that each focus on one or two key messages, with a recommended maximum of 10 sections in the release. The whole release should take no more than 10 minutes to read. Our analytics app contains insights on how long it takes the average user to read your release.\nTo keep the release short only include information if there is something interesting to say - the commentary is there to tell a story, people looking for specific figures will use the table tool, or download the underlying data instead. Do not try to summarise every number in the commentary.\nAvoid having large blocks of text as they are hard to read and users scan them and miss the detail. Graphs and tables break up the content but only include these where they add value; you do not need a graph or chart in every section.\nUse plain English and shorter sentences, aim for an average of 15-20 words per sentence. Do not overload sentences with numbers and avoid ‘elevator commentary’ that describes small movements in the whole series without giving any insight (use a summary table instead if it is interesting, or leave it out entirely). Be impartial and objective; avoid using sensationalist terms or terms that reflect a judgement such as “very few” “only” or “strong”.\nExplain complex concepts in plain English within the text. Remember that for many of our users, confidence intervals and significant differences are complex concepts that need explaining.\n\nDo not use footnotes in the text of your content. They’re designed for reference in print, not web pages. Always consider the user need first. If the information in the footnotes is important, include it in the body text. If it’s not, leave it out."
+ "objectID": "CODE_OF_CONDUCT.html#enforcement-responsibilities",
+ "href": "CODE_OF_CONDUCT.html#enforcement-responsibilities",
+ "title": "Contributor Covenant Code of Conduct",
+ "section": "",
+ "text": "Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.\nCommunity leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate."
},
{
- "objectID": "writing-visualising/cd.html#writing-about-characteristics",
- "href": "writing-visualising/cd.html#writing-about-characteristics",
- "title": "Writing and visualising",
- "section": "Writing about characteristics",
- "text": "Writing about characteristics\nThere is a wide range of guidance available from the GSS, ONS and the Cabinet Office around writing about characteristics. Statistics content published on EES should adhere to the principles outlined by the above. The data harmonisation champions group are in the process of collating the most recent guidance from all these sources and summarising it below.\nIf you need some steer on how to report on a particular characteristics, the below links provide some useful starting points:\n\nAnalysis Function Data Harmonisation\nCabinet Office guidance on writing about statistics\n\nYou can also get in touch with the DfE Data Harmonisation Champions Group via statistics.development@education.gov.uk.\n\n\nWriting about ethnicity\n\nFor the official names of ethnicity filters to use in data files, please check our guidance on creating statistics. The below outlines some key points in writing about ethnicity in publication content.\n\n\nEthnic minorities and not BAME\n\nAs outlined in the CRED report, grouping ethnicity at a higher level than the 5 major ethnic groups should be avoided as it risks masking the true disparities between, and/or life experiences of, people from different ethnic groups.\nThere will, however, be specific cases where data quality or comparisons with past metrics does not allow for statistics to be produced at the level of the major or minor ethnic groups. In these cases, the group ‘ethnic minorities’ should be used exclusively to refer to all ethnic groups except the White British group, this includes White minorities, such as Gypsy, Roma and Irish Traveller groups.\nFor comparisons with the White group as a whole, ‘all other ethnic groups combined’ or ‘ethnic minorities (excluding White minorities)’ should be used. It may also be necessary to refer to ‘White’ and ‘other than White’ if space is limited, but this should be avoided if possible.\nDo not use the terms ‘non-White’, BAME (Black, Asian, and Minority Ethnic), or BME (Black and minority ethnic). These terms can be perceived as demeaning and can be confusing as to what groups are being included and excluded. For comparison with past metrics, BAME and BME should be replaced by the term “ethnic minorities (excluding White minorities)” unless White minorities were specifically mentioned.\n\n\n\nOrdering of groups\n\nThere are several methods of ordering the ethnic groups, depending on where and how they are used:\n\nAlphabetical: use in tables and when listing ethnic groups (with ‘Other’ and sometimes ‘Unknown’ as a final category)\nIn expected order of size (with largest first): useful in charts and visualisations as it makes data and patterns easier to read"
+ "objectID": "CODE_OF_CONDUCT.html#scope",
+ "href": "CODE_OF_CONDUCT.html#scope",
+ "title": "Contributor Covenant Code of Conduct",
+ "section": "",
+ "text": "This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event."
},
{
- "objectID": "writing-visualising/cd.html#reporting-and-interpreting-numbers",
- "href": "writing-visualising/cd.html#reporting-and-interpreting-numbers",
- "title": "Writing and visualising",
- "section": "Reporting and interpreting numbers",
- "text": "Reporting and interpreting numbers\nReporting the numbers you work with is an important first step toward writing effective numeric descriptions. By including numbers in text, table or chart, you give the user the raw materials with which to perform comparisons across time, places or groups. However, if you stop there, you leave it to your readers to figure out how those data answer the question at hand.\nPoor: “In 2010, there were 20, 370 overall homicides related to crime, 13,000 which were related to gun incidents, 7,370 related to other weapons. In 2020, they were 18,900 overall homicides, 11,000 which were related to gun incidents, 9,900 related to other weapons (Figure 1)”.\nThe description above simply lists statistics from charts without explaining how they relate to one another or how the statistics address the initial question in the opening paragraph.\nBetter: “The total number of homicides rose until the mid-2000s and then declined until 2020. As shown in Figure 1, the increase and subsequent decrease in homicides were driven by trends in gun-related homicides. In 2020, there was roughly 1.5 times as many homicides were committed with guns as with other types of weapons (11,000 versus 7265; Figure 1); whereas in 2010, roughly 2 times as many homicides were committed with guns versus other weapons, 13000 and 6500, respectively. Figure 2 examines whether gun-related homicides showed the same time trend among all age groups. As shown in Figure 2 in the two youngest groups of offenders, gun-related homicides increased substantially between 2000 and 2010, and then decreased steadily until 2020. In contrast, the number of gun related homicides committed by older offenders decreased slowly throughout the time-period shown”.\nTry to use prose to summarise the patterns so your user can see the general relationship in the table or chart – the forest not the individual trees. Try not to report every number from the table or chart or pick a few arbitrary numbers to contrast in sentence form without considering whether or not those numbers represent an underlying general pattern. Paint the big picture rather than reiterating all the little details. This will help you tell a clear story with numbers as evidence."
+ "objectID": "CODE_OF_CONDUCT.html#enforcement",
+ "href": "CODE_OF_CONDUCT.html#enforcement",
+ "title": "Contributor Covenant Code of Conduct",
+ "section": "",
+ "text": "Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at statistics.development@education.gov.uk. All complaints will be reviewed and investigated promptly and fairly.\nAll community leaders are obligated to respect the privacy and security of the reporter of any incident."
},
{
- "objectID": "writing-visualising/cd.html#creating-hyperlinks",
- "href": "writing-visualising/cd.html#creating-hyperlinks",
- "title": "Writing and visualising",
- "section": "Creating hyperlinks",
- "text": "Creating hyperlinks\nAvoid using full URLs in text. Hyperlinks should be used and they should provide a clear description of the destination. Avoid using ‘For more information click here’. Screen readers often collate all links on a page into one list, so having numerous ‘click here’ links listed is confusing to the user and gives no description of the destinations. ‘For more information see Guidance to support the summer 2021 exams’ is an example of a good hyperlink.\nBest practice on creating hyperlinks, particularly how to name them and common pitfalls to avoid can be found on this introduction to html guide. Those of us using EES don’t need to worry about writing the raw html for the anchor links, and should instead focus on the section referring to how to name and title links."
+ "objectID": "CODE_OF_CONDUCT.html#enforcement-guidelines",
+ "href": "CODE_OF_CONDUCT.html#enforcement-guidelines",
+ "title": "Contributor Covenant Code of Conduct",
+ "section": "",
+ "text": "Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:\n\n\nCommunity Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.\nConsequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.\n\n\n\nCommunity Impact: A violation through a single incident or series of actions.\nConsequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.\n\n\n\nCommunity Impact: A serious violation of community standards, including sustained inappropriate behavior.\nConsequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.\n\n\n\nCommunity Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.\nConsequence: A permanent ban from any sort of public interaction within the community."
},
{
- "objectID": "writing-visualising/cd.html#search-engine-optimisation",
- "href": "writing-visualising/cd.html#search-engine-optimisation",
- "title": "Writing and visualising",
- "section": "Search engine optimisation",
- "text": "Search engine optimisation\nSearch engine optimisation (SEO) makes it easier for users to find your data through search engines like Google. Some top tips include:\n\nKeeping your release title shorter than 50-60 characters. This means the full title can be displayed on the search engine results\nAvoid listing key words: search engines penalise anything not recognised as a full sentence.\nMake use of our analytics app to explore what your users are doing: what accordions are they clicking on, what are they searching for? This could give an idea of what content you should focus on in future, and which areas are no longer of interest to most of your users.\n\nFollowing best practice in writing about statistics is of increasing importance. As shown in the below example any sentence could be pulled out into a snippet and shown in a search engine to users who are searching for related information:\n\nWe should make a concerted effort to ensure that we are answering the questions people are interested in as search engines are getting smarter and pulling this information directly out of webpages. See the following example of a google search using a snippet from one of our publications as an answer in the search engine results itself:"
+ "objectID": "CODE_OF_CONDUCT.html#attribution",
+ "href": "CODE_OF_CONDUCT.html#attribution",
+ "title": "Contributor Covenant Code of Conduct",
+ "section": "",
+ "text": "This Code of Conduct is adapted from the Contributor Covenant, version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.\nCommunity Impact Guidelines were inspired by Mozilla’s code of conduct enforcement ladder.\nFor answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations."
},
{
- "objectID": "writing-visualising/cd.html#visualisation-titles",
- "href": "writing-visualising/cd.html#visualisation-titles",
- "title": "Writing and visualising",
- "section": "Visualisation titles",
- "text": "Visualisation titles\nFor tables and charts, active titles are descriptive and tell the trend by highlighting the main story. They should be short - aim for 10 words or less and avoid going over more than one line. Active titles give users the main message without having to find the text that accompanies the chart, and makes it easier for journalists to use your chart directly without having to write their own summary.\nPut information such as the measure, source, population, geographical coverage and time period in a caption if they are not obvious from the chart content. Add further context and information in the main text. Remember that tables and charts should be usable even if isolated from the rest of the release."
+ "objectID": "index.html",
+ "href": "index.html",
+ "title": "Analysts’s Guide",
+ "section": "",
+ "text": "This website is a guide aimed at anyone working in analysis or statistics in the Department for Education (DfE). It includes tips on best practice and learning resources across a number of areas.\nWe hope it can prove a useful community driven resource for everyone from the most experienced analyst right through to those just starting out. If you have any feedback, suggested additions, or wish to challenge any of the guidance, feel free to use the GitHub links to suggest changes directly, or contact us at the email you can find at the bottom of the page."
},
{
- "objectID": "writing-visualising/cd.html#using-data-blocks-effectively",
- "href": "writing-visualising/cd.html#using-data-blocks-effectively",
- "title": "Writing and visualising",
- "section": "Using data blocks effectively",
- "text": "Using data blocks effectively\nAim for fewer tables, and keep them small. As a guide, we suggest no more than one table per accordion section.\nPresentation tables are the tables you include within the accordion sections of your release to quickly visualise numbers. Unlike the underlying data files, the presentation tables focus on specific parts of the story you are telling. They are distinct from, and should never be a copy of an underlying data file.\nYou should only be including small tables sparingly where they add value to your release and not use them as a straight copy of the ready-made Excel tables previously published on gov.uk. The data itself is there for users to access through other means, therefore any data in the commentary should only be there if it is enhancing the story.\n\nYou do not need to recreate all of the old excel tables, users can find the numbers they are interested in using the table tool, or analyse the underlying data if they want the data behind the release."
+ "objectID": "index.html#learning-and-development",
+ "href": "index.html#learning-and-development",
+ "title": "Analysts’s Guide",
+ "section": "Learning and development",
+ "text": "Learning and development\nGeneral resources - Useful learning resources, and support to get you started\nSQL - Guidance and tips for accessing data via databases with SQL\nR - Guidance and tips for using R\nGit - Guidance and tips for version control with Git\nPython - Guidance and tips for using Python [NEW PAGE]"
},
{
- "objectID": "writing-visualising/cd.html#charts-in-ees",
- "href": "writing-visualising/cd.html#charts-in-ees",
- "title": "Writing and visualising",
- "section": "Charts in EES",
- "text": "Charts in EES\nAny charts in your release should be appropriate for the data, reinforcing the key messages. For more detailed information on how to build charts in EES, visit our EES charts guidance section.\nAny analyst building charts in or outside of EES should make themselves familiar with the [GSS data visualisation guidance](GSS colours in visualisations guidance.\nMore complex charts cannot currently be built in EES. You should only use complex charts where there is a clear user need, as simple charts are the easiest for users to understand. Please see the ONS guidance for visualisations or contact the Statstics Development Team if you’re considering a more complex chart built outside of EES.\nDWP have created a Data Visualisation Thinking course that may be useful to look at when creating more complex charts."
+ "objectID": "index.html#statistics-production",
+ "href": "index.html#statistics-production",
+ "title": "Analysts’s Guide",
+ "section": "Statistics production",
+ "text": "Statistics production\nHow to publish - Guidance for how to publish different types of statistics\nOpen data standards - Guidance on how to structure data files\nPublication pipelines - Link to the RAP for stats? [NEW PAGE / REDIRECT]\nUsing EES - How to use the features in Explore Education Statistics\nGood examples in EES - Examples of good practice in Explore Education Statistics\nEmbedded visualisations in EES - How to embed R-Shiny charts in EES publications\nUser engagement - Guidance on understanding and engaging with the users of published statistics\nEES analytics - Understanding how users are interacting with your publications"
},
{
- "objectID": "writing-visualising/cd.html#colours-in-charts",
- "href": "writing-visualising/cd.html#colours-in-charts",
- "title": "Writing and visualising",
- "section": "Colours in charts",
- "text": "Colours in charts\nThe most important consideration when using colour is to avoid relying on it for interpretation. It should be seen as an enhancement, and your charts should be understandable without it.\nFor any charts made outside of EES you should use the suggested colours from the GSS colours in visualisations guidance. The following sections show the colours that are available for charts in EES and suggest the way to use them with one another, more detail on why can be found in the GSS guidance itself.\nWe strongly recommend you stick to the codes as outlined, if you can’t for any reason, please make sure you follow the guidance on developing your own colour palette and get in touch with us so we can understand why.\n\n\nCategorical colour palette\n\nCategorical data can be divided into groups or categories by using names or labels. This palette has four colours. We recommend a limit of four categories as best practice for basic data visualisations. The ordering of the palette is important as not all colours are accessible when paired together.\n\n\n\nColour name\nHex code\nRGB\nCYMK\n\n\n\n\nDark blue\n#12436D\n18, 67, 109\n36, 16, 0, 57\n\n\nTurquoise\n#28A197\n40, 161, 151\n75, 0, 6, 37\n\n\nDark pink\n#801650\n128, 22, 80\n0, 83, 38, 50\n\n\nOrange\n#F46A25\n244, 106, 37\n0, 57, 85, 4\n\n\n\n\n\n\nSequential colour palette\n\nSequential data is any sort of data where the order of series has some meaning. For example, age groups ascending in age. This palette has three colours and even this pushes the boundaries for contrast ratios. Any charts made with shades of a specific hue should be accessible without colour.\n\n\n\nColour name\nHex code\nRGB\nCYMK\n\n\n\n\nDark blue\n#12436D\n18, 67, 109\n36, 16, 0, 57\n\n\nMid blue\n#2073BC\n32, 115, 188\n0, 57, 85, 4\n\n\nLight blue\n#6BACE6\n107, 172, 230\n75, 0, 6, 37\n\n\n\n\n\n\nFocus palette\n\nOn focus charts, colour is used to highlight specific elements to help users understand the information. One line out of many will be highlighted as a colour and the rest of the series remain grey.\n\n\n\nColour name\nHex code\nRGB\nCYMK\n\n\n\n\nDark blue\n#12436D\n18, 67, 109\n36, 16, 0, 57\n\n\nGrey\n#BFBFBF\n191, 191, 191\n0, 0, 0, 25\n\n\n\n\n\n\nLarger palettes\n\nAll charts should be made as simple as possible so that the message is easy to understand and interpret.\nIf you do want to use additional colours you need to ensure that the chart is understandable without colour as no series with more than four colours will ever be fully accessible alone.\nFor categorical data, we recommend the following extensions to the categorical palette. Once again, the ordering is important. If you’re making a chart in EES these are not default colours, so you will need to use the colour picker to specify them.\n\n\n\nColour name\nHex code\nRGB\nCYMK\n\n\n\n\nDark blue\n#12436D\n18, 67, 109\n36, 16, 0, 57\n\n\nTurquoise\n#28A197\n40, 161, 151\n75, 0, 6, 37\n\n\nDark pink\n#801650\n128, 22, 80\n0, 83, 38, 50\n\n\nOrange\n#F46A25\n244, 106, 37\n0, 57, 85, 4\n\n\nDark grey\n#3D3D3D\n61, 61, 61\n0, 0, 0, 76\n\n\nLight purple\n#A285D1\n162, 133, 209\n22, 36, 0, 18\n\n\n\nFor larger palettes when using sequential data, we recommend using ColorBrewer to find the hex codes for multiple shades of a given hue."
+ "objectID": "index.html#writing-and-visualising-data",
+ "href": "index.html#writing-and-visualising-data",
+ "title": "Analysts’s Guide",
+ "section": "Writing and visualising data",
+ "text": "Writing and visualising data\nPublic dashboards - Guidance for publishing public facing statistics dashboards\nWriting about statistics - Things to consider when writing about statistics and data\nVisualising data - Resources and best practice to guide you when visualising data [NEW PAGE]"
},
{
- "objectID": "writing-visualising/cd.html#visualising-breaks-in-a-timeseries",
- "href": "writing-visualising/cd.html#visualising-breaks-in-a-timeseries",
- "title": "Writing and visualising",
- "section": "Visualising breaks in a timeseries",
- "text": "Visualising breaks in a timeseries\nWe recommend including any missing data from breaks in a time series in your data file using the appropriate GSS symbol, such as in this example table:\n\n\n\n\n\n\n\n\n\n\n\n\n2013/14\n2014/15\n2015/16\n2016/17\n2017/18\n\n\n\n\nNumber of pupil enrollments\n3,627,985\n3,713,774\n3,796,146\nx\n3,885,774\n\n\nNumber of schools\n16,705\n16,723\n16,736\nx\n16,739\n\n\n\n\nThere may be times when including missing data increases the file size too much, or becomes unwieldy, if you’re unsure and would like advice on your data contact statistics.development@education.gov.uk.\n\nBy including the missing data in your open data files you can then create charts in EES that represent this. Start off by creating a data block with the data you want to build the chart from.\n\nOn the chart configuration tab there is a toggle for visualising non-numeric values in the data as 0. On the first chart configuration tab, there’s a check box that will toggle between showing and hiding them.\n\nWhen selected, you will then see that this data appears in the chart as if the indicator values are 0."
+ "objectID": "index.html#reproducible-analytical-pipelines-rap",
+ "href": "index.html#reproducible-analytical-pipelines-rap",
+ "title": "Analysts’s Guide",
+ "section": "Reproducible Analytical Pipelines (RAP)",
+ "text": "Reproducible Analytical Pipelines (RAP)\nRAP expectations - Guidance for all analysts on expectations of RAP\nRAP support - Details on support available for RAP in DfE\nRAP FAQs - Frequently asked questions about RAP\nRAP in statistics - Detailed RAP guidance for statistics publications"
},
{
- "objectID": "writing-visualising/cd.html#glossary",
- "href": "writing-visualising/cd.html#glossary",
- "title": "Writing and visualising",
- "section": "Glossary",
- "text": "Glossary\nThe glossary in Explore Education Statistics is intended for definitions used across all publications. Please send any definitions you would like adding to the glossary to the Statistics Development Team.\nOnce definitions have been added, you can add them as URL links into your commentary. E.g. to see the definition for ‘Otherwise vulnerable’, you’d make that phrase a hyperlink, and use https://explore-education-statistics.service.gov.uk/glossary#otherwise-vulnerable as the link.\nBefore creating a new definition, check the glossary to see if one is already there. When creating glossary entries you should use standard DfE definitions, only write your own if it is something that relates specifically to your statistics. Do not use technical terms in your definition, stick to plain English.\nInclude essential definitions in the commentary where they are relevant, you should not expect users to go to the glossary or about these statistics section for information they need to understand the statistics. Do not use the glossary to explain methodology such as classification decisions, explain these in the text if essential or otherwise in the methodology document."
+ "objectID": "index.html#ada-and-databricks",
+ "href": "index.html#ada-and-databricks",
+ "title": "Analysts’s Guide",
+ "section": "ADA and databricks",
+ "text": "ADA and databricks\nSomething - ? TBC"
},
{
- "objectID": "writing-visualising/cd.html#publication-scrums",
- "href": "writing-visualising/cd.html#publication-scrums",
- "title": "Writing and visualising",
- "section": "Publication scrums",
- "text": "Publication scrums\nThe Statistics Services Unit organise regular publication scrums, which enable publications to be looked at with fresh eyes through a workshop with volunteers and the publication team. The content checklist, which breaks down the components of producing Best Practice content into smaller chunks with prompt questions is used (Download content checklist), along with workshop slides.\nIf you are responsible for signing off publications, then please download and see the Statistics Leadership Group paper highlighting top tips for content design.\nOne of the early outcomes from running publication scrums, was that great content design is more of an art form than science and the scrum process was found to be really useful in enabling a wider element of scrutiny and friendly challenge, in an enjoyable environment to consider publication content.\nYou can experience how a scrum runs through watching this scrum-along (features the scrum up until groups breakout to discuss different elements) and supporting slides\n\n\n\n\nA full before / after scrum along is available to watch, with slides available to download separately.\nFor a walk through of some of the end to end benefits the scrums have had, take a look at this video for Schools and pupils Statistics Team.\nFinally, an example of a team who has been through the scrum process for 3 publications talk through their realised benefits, showcasing the type of benefits potentially others could also realise is also available to watch on sharepoint.\nIf you’re interested in being involved in future scrums, please get in contact us."
+ "objectID": "index.html#contact-us",
+ "href": "index.html#contact-us",
+ "title": "Analysts’s Guide",
+ "section": "Contact us",
+ "text": "Contact us\nOur mailbox is always monitored and is available for anyone in DfE to ask questions about statistics, whether that is about RAP, building dashboards, coding support, learning and development or statistics publications.\n\nstatistics.development@education.gov.uk\n9am-5pm, Monday-Friday, aim to reply within 1-2 days"
}
]
\ No newline at end of file
diff --git a/site_libs/bootstrap/bootstrap-dark.min.css b/site_libs/bootstrap/bootstrap-dark.min.css
index 5f08ad4..134514a 100644
--- a/site_libs/bootstrap/bootstrap-dark.min.css
+++ b/site_libs/bootstrap/bootstrap-dark.min.css
@@ -7,4 +7,4 @@
*
* ansi colors from IPython notebook's
*
-*/.ansi-black-fg{color:#3e424d}.ansi-black-bg{background-color:#3e424d}.ansi-black-intense-fg{color:#282c36}.ansi-black-intense-bg{background-color:#282c36}.ansi-red-fg{color:#e75c58}.ansi-red-bg{background-color:#e75c58}.ansi-red-intense-fg{color:#b22b31}.ansi-red-intense-bg{background-color:#b22b31}.ansi-green-fg{color:#00a250}.ansi-green-bg{background-color:#00a250}.ansi-green-intense-fg{color:#007427}.ansi-green-intense-bg{background-color:#007427}.ansi-yellow-fg{color:#ddb62b}.ansi-yellow-bg{background-color:#ddb62b}.ansi-yellow-intense-fg{color:#b27d12}.ansi-yellow-intense-bg{background-color:#b27d12}.ansi-blue-fg{color:#208ffb}.ansi-blue-bg{background-color:#208ffb}.ansi-blue-intense-fg{color:#0065ca}.ansi-blue-intense-bg{background-color:#0065ca}.ansi-magenta-fg{color:#d160c4}.ansi-magenta-bg{background-color:#d160c4}.ansi-magenta-intense-fg{color:#a03196}.ansi-magenta-intense-bg{background-color:#a03196}.ansi-cyan-fg{color:#60c6c8}.ansi-cyan-bg{background-color:#60c6c8}.ansi-cyan-intense-fg{color:#258f8f}.ansi-cyan-intense-bg{background-color:#258f8f}.ansi-white-fg{color:#c5c1b4}.ansi-white-bg{background-color:#c5c1b4}.ansi-white-intense-fg{color:#a1a6b2}.ansi-white-intense-bg{background-color:#a1a6b2}.ansi-default-inverse-fg{color:#fff}.ansi-default-inverse-bg{background-color:#000}.ansi-bold{font-weight:bold}.ansi-underline{text-decoration:underline}:root{--quarto-body-bg: #fff;--quarto-body-color: #333;--quarto-text-muted: #6c757d;--quarto-border-color: #dee2e6;--quarto-border-width: 1px;--quarto-border-radius: 0.25rem}table.gt_table{color:var(--quarto-body-color);font-size:1em;width:100%;background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_column_spanner_outer{color:var(--quarto-body-color);background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_col_heading{color:var(--quarto-body-color);font-weight:bold;background-color:rgba(0,0,0,0)}table.gt_table thead.gt_col_headings{border-bottom:1px solid currentColor;border-top-width:inherit;border-top-color:var(--quarto-border-color)}table.gt_table thead.gt_col_headings:not(:first-child){border-top-width:1px;border-top-color:var(--quarto-border-color)}table.gt_table td.gt_row{border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-width:0px}table.gt_table tbody.gt_table_body{border-top-width:1px;border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-color:currentColor}div.columns{display:initial;gap:initial}div.column{display:inline-block;overflow-x:initial;vertical-align:top;width:50%}.code-annotation-tip-content{word-wrap:break-word}.code-annotation-container-hidden{display:none !important}dl.code-annotation-container-grid{display:grid;grid-template-columns:min-content auto}dl.code-annotation-container-grid dt{grid-column:1}dl.code-annotation-container-grid dd{grid-column:2}pre.sourceCode.code-annotation-code{padding-right:0}code.sourceCode .code-annotation-anchor{z-index:100;position:absolute;right:.5em;left:inherit;background-color:rgba(0,0,0,0)}:root{--mermaid-bg-color: #fff;--mermaid-edge-color: #adb5bd;--mermaid-node-fg-color: #333;--mermaid-fg-color: #333;--mermaid-fg-color--lighter: #4d4d4d;--mermaid-fg-color--lightest: #666666;--mermaid-font-family: Ubuntu, -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol;--mermaid-label-bg-color: #fff;--mermaid-label-fg-color: #e95420;--mermaid-node-bg-color: rgba(233, 84, 32, 0.1);--mermaid-node-fg-color: #333}@media print{:root{font-size:11pt}#quarto-sidebar,#TOC,.nav-page{display:none}.page-columns .content{grid-column-start:page-start}.fixed-top{position:relative}.panel-caption,.figure-caption,figcaption{color:#666}}.code-copy-button{position:absolute;top:0;right:0;border:0;margin-top:5px;margin-right:5px;background-color:rgba(0,0,0,0);z-index:3}.code-copy-button:focus{outline:none}.code-copy-button-tooltip{font-size:.75em}.code-copy-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}.code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}.code-copy-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}.code-copy-button-checked:hover>.bi::before{background-image:url('data:image/svg+xml,')}main ol ol,main ul ul,main ol ul,main ul ol{margin-bottom:1em}ul>li:not(:has(>p))>ul,ol>li:not(:has(>p))>ul,ul>li:not(:has(>p))>ol,ol>li:not(:has(>p))>ol{margin-bottom:0}ul>li:not(:has(>p))>ul>li:has(>p),ol>li:not(:has(>p))>ul>li:has(>p),ul>li:not(:has(>p))>ol>li:has(>p),ol>li:not(:has(>p))>ol>li:has(>p){margin-top:1rem}body{margin:0}main.page-columns>header>h1.title,main.page-columns>header>.title.h1{margin-bottom:0}@media(min-width: 992px){body .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.fullcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] 35px [page-end-inset page-end] 5fr [screen-end-inset] 1.5em}body.slimcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.listing:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 3em [body-end] 50px [body-end-outset] minmax(0px, 250px) [page-end-inset] minmax(50px, 100px) [page-end] 1fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 175px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 175px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] minmax(25px, 50px) [page-start-inset] minmax(50px, 150px) [body-start-outset] minmax(25px, 50px) [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] minmax(25px, 50px) [body-end-outset] minmax(50px, 150px) [page-end-inset] minmax(25px, 50px) [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(50px, 100px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 50px [page-start-inset] minmax(50px, 150px) [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(450px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 50px [page-start-inset] minmax(50px, 150px) [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(450px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(50px, 150px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] minmax(25px, 50px) [page-start-inset] minmax(50px, 150px) [body-start-outset] minmax(25px, 50px) [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] minmax(25px, 50px) [body-end-outset] minmax(50px, 150px) [page-end-inset] minmax(25px, 50px) [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}}@media(max-width: 991.98px){body .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.fullcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.slimcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.listing:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 1250px - 3em )) [body-content-end body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 145px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 145px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1.5em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(75px, 150px) [page-end-inset] 25px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 4fr [screen-end-inset] 1.5em [screen-end]}body.docked.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 4fr [screen-end-inset] 1.5em [screen-end]}body.floating.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(75px, 150px) [page-end-inset] 25px [page-end] 4fr [screen-end-inset] 1.5em [screen-end]}}@media(max-width: 767.98px){body .page-columns,body.fullcontent:not(.floating):not(.docked) .page-columns,body.slimcontent:not(.floating):not(.docked) .page-columns,body.docked .page-columns,body.docked.slimcontent .page-columns,body.docked.fullcontent .page-columns,body.floating .page-columns,body.floating.slimcontent .page-columns,body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}nav[role=doc-toc]{display:none}}body,.page-row-navigation{grid-template-rows:[page-top] max-content [contents-top] max-content [contents-bottom] max-content [page-bottom]}.page-rows-contents{grid-template-rows:[content-top] minmax(max-content, 1fr) [content-bottom] minmax(60px, max-content) [page-bottom]}.page-full{grid-column:screen-start/screen-end !important}.page-columns>*{grid-column:body-content-start/body-content-end}.page-columns.column-page>*{grid-column:page-start/page-end}.page-columns.column-page-left>*{grid-column:page-start/body-content-end}.page-columns.column-page-right>*{grid-column:body-content-start/page-end}.page-rows{grid-auto-rows:auto}.header{grid-column:screen-start/screen-end;grid-row:page-top/contents-top}#quarto-content{padding:0;grid-column:screen-start/screen-end;grid-row:contents-top/contents-bottom}body.floating .sidebar.sidebar-navigation{grid-column:page-start/body-start;grid-row:content-top/page-bottom}body.docked .sidebar.sidebar-navigation{grid-column:screen-start/body-start;grid-row:content-top/page-bottom}.sidebar.toc-left{grid-column:page-start/body-start;grid-row:content-top/page-bottom}.sidebar.margin-sidebar{grid-column:body-end/page-end;grid-row:content-top/page-bottom}.page-columns .content{grid-column:body-content-start/body-content-end;grid-row:content-top/content-bottom;align-content:flex-start}.page-columns .page-navigation{grid-column:body-content-start/body-content-end;grid-row:content-bottom/page-bottom}.page-columns .footer{grid-column:screen-start/screen-end;grid-row:contents-bottom/page-bottom}.page-columns .column-body{grid-column:body-content-start/body-content-end}.page-columns .column-body-fullbleed{grid-column:body-start/body-end}.page-columns .column-body-outset{grid-column:body-start-outset/body-end-outset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset table{background:#fff}.page-columns .column-body-outset-left{grid-column:body-start-outset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset-left table{background:#fff}.page-columns .column-body-outset-right{grid-column:body-content-start/body-end-outset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset-right table{background:#fff}.page-columns .column-page{grid-column:page-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page table{background:#fff}.page-columns .column-page-inset{grid-column:page-start-inset/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset table{background:#fff}.page-columns .column-page-inset-left{grid-column:page-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset-left table{background:#fff}.page-columns .column-page-inset-right{grid-column:body-content-start/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset-right figcaption table{background:#fff}.page-columns .column-page-left{grid-column:page-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-left table{background:#fff}.page-columns .column-page-right{grid-column:body-content-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-right figcaption table{background:#fff}#quarto-content.page-columns #quarto-margin-sidebar,#quarto-content.page-columns #quarto-sidebar{z-index:1}@media(max-width: 991.98px){#quarto-content.page-columns #quarto-margin-sidebar.collapse,#quarto-content.page-columns #quarto-sidebar.collapse,#quarto-content.page-columns #quarto-margin-sidebar.collapsing,#quarto-content.page-columns #quarto-sidebar.collapsing{z-index:1055}}#quarto-content.page-columns main.column-page,#quarto-content.page-columns main.column-page-right,#quarto-content.page-columns main.column-page-left{z-index:0}.page-columns .column-screen-inset{grid-column:screen-start-inset/screen-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset table{background:#fff}.page-columns .column-screen-inset-left{grid-column:screen-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-left table{background:#fff}.page-columns .column-screen-inset-right{grid-column:body-content-start/screen-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-right table{background:#fff}.page-columns .column-screen{grid-column:screen-start/screen-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen table{background:#fff}.page-columns .column-screen-left{grid-column:screen-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-left table{background:#fff}.page-columns .column-screen-right{grid-column:body-content-start/screen-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-right table{background:#fff}.page-columns .column-screen-inset-shaded{grid-column:screen-start/screen-end;padding:1em;background:#e9ecef;z-index:998;transform:translate3d(0, 0, 0);margin-bottom:1em}.zindex-content{z-index:998;transform:translate3d(0, 0, 0)}.zindex-modal{z-index:1055;transform:translate3d(0, 0, 0)}.zindex-over-content{z-index:999;transform:translate3d(0, 0, 0)}img.img-fluid.column-screen,img.img-fluid.column-screen-inset-shaded,img.img-fluid.column-screen-inset,img.img-fluid.column-screen-inset-left,img.img-fluid.column-screen-inset-right,img.img-fluid.column-screen-left,img.img-fluid.column-screen-right{width:100%}@media(min-width: 992px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-end/page-end !important;z-index:998}.column-sidebar{grid-column:page-start/body-start !important;z-index:998}.column-leftmargin{grid-column:screen-start-inset/body-start !important;z-index:998}.no-row-height{height:1em;overflow:visible}}@media(max-width: 991.98px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-end/page-end !important;z-index:998}.no-row-height{height:1em;overflow:visible}.page-columns.page-full{overflow:visible}.page-columns.toc-left .margin-caption,.page-columns.toc-left div.aside,.page-columns.toc-left aside,.page-columns.toc-left .column-margin{grid-column:body-content-start/body-content-end !important;z-index:998;transform:translate3d(0, 0, 0)}.page-columns.toc-left .no-row-height{height:initial;overflow:initial}}@media(max-width: 767.98px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-content-start/body-content-end !important;z-index:998;transform:translate3d(0, 0, 0)}.no-row-height{height:initial;overflow:initial}#quarto-margin-sidebar{display:none}#quarto-sidebar-toc-left{display:none}.hidden-sm{display:none}}.panel-grid{display:grid;grid-template-rows:repeat(1, 1fr);grid-template-columns:repeat(24, 1fr);gap:1em}.panel-grid .g-col-1{grid-column:auto/span 1}.panel-grid .g-col-2{grid-column:auto/span 2}.panel-grid .g-col-3{grid-column:auto/span 3}.panel-grid .g-col-4{grid-column:auto/span 4}.panel-grid .g-col-5{grid-column:auto/span 5}.panel-grid .g-col-6{grid-column:auto/span 6}.panel-grid .g-col-7{grid-column:auto/span 7}.panel-grid .g-col-8{grid-column:auto/span 8}.panel-grid .g-col-9{grid-column:auto/span 9}.panel-grid .g-col-10{grid-column:auto/span 10}.panel-grid .g-col-11{grid-column:auto/span 11}.panel-grid .g-col-12{grid-column:auto/span 12}.panel-grid .g-col-13{grid-column:auto/span 13}.panel-grid .g-col-14{grid-column:auto/span 14}.panel-grid .g-col-15{grid-column:auto/span 15}.panel-grid .g-col-16{grid-column:auto/span 16}.panel-grid .g-col-17{grid-column:auto/span 17}.panel-grid .g-col-18{grid-column:auto/span 18}.panel-grid .g-col-19{grid-column:auto/span 19}.panel-grid .g-col-20{grid-column:auto/span 20}.panel-grid .g-col-21{grid-column:auto/span 21}.panel-grid .g-col-22{grid-column:auto/span 22}.panel-grid .g-col-23{grid-column:auto/span 23}.panel-grid .g-col-24{grid-column:auto/span 24}.panel-grid .g-start-1{grid-column-start:1}.panel-grid .g-start-2{grid-column-start:2}.panel-grid .g-start-3{grid-column-start:3}.panel-grid .g-start-4{grid-column-start:4}.panel-grid .g-start-5{grid-column-start:5}.panel-grid .g-start-6{grid-column-start:6}.panel-grid .g-start-7{grid-column-start:7}.panel-grid .g-start-8{grid-column-start:8}.panel-grid .g-start-9{grid-column-start:9}.panel-grid .g-start-10{grid-column-start:10}.panel-grid .g-start-11{grid-column-start:11}.panel-grid .g-start-12{grid-column-start:12}.panel-grid .g-start-13{grid-column-start:13}.panel-grid .g-start-14{grid-column-start:14}.panel-grid .g-start-15{grid-column-start:15}.panel-grid .g-start-16{grid-column-start:16}.panel-grid .g-start-17{grid-column-start:17}.panel-grid .g-start-18{grid-column-start:18}.panel-grid .g-start-19{grid-column-start:19}.panel-grid .g-start-20{grid-column-start:20}.panel-grid .g-start-21{grid-column-start:21}.panel-grid .g-start-22{grid-column-start:22}.panel-grid .g-start-23{grid-column-start:23}@media(min-width: 576px){.panel-grid .g-col-sm-1{grid-column:auto/span 1}.panel-grid .g-col-sm-2{grid-column:auto/span 2}.panel-grid .g-col-sm-3{grid-column:auto/span 3}.panel-grid .g-col-sm-4{grid-column:auto/span 4}.panel-grid .g-col-sm-5{grid-column:auto/span 5}.panel-grid .g-col-sm-6{grid-column:auto/span 6}.panel-grid .g-col-sm-7{grid-column:auto/span 7}.panel-grid .g-col-sm-8{grid-column:auto/span 8}.panel-grid .g-col-sm-9{grid-column:auto/span 9}.panel-grid .g-col-sm-10{grid-column:auto/span 10}.panel-grid .g-col-sm-11{grid-column:auto/span 11}.panel-grid .g-col-sm-12{grid-column:auto/span 12}.panel-grid .g-col-sm-13{grid-column:auto/span 13}.panel-grid .g-col-sm-14{grid-column:auto/span 14}.panel-grid .g-col-sm-15{grid-column:auto/span 15}.panel-grid .g-col-sm-16{grid-column:auto/span 16}.panel-grid .g-col-sm-17{grid-column:auto/span 17}.panel-grid .g-col-sm-18{grid-column:auto/span 18}.panel-grid .g-col-sm-19{grid-column:auto/span 19}.panel-grid .g-col-sm-20{grid-column:auto/span 20}.panel-grid .g-col-sm-21{grid-column:auto/span 21}.panel-grid .g-col-sm-22{grid-column:auto/span 22}.panel-grid .g-col-sm-23{grid-column:auto/span 23}.panel-grid .g-col-sm-24{grid-column:auto/span 24}.panel-grid .g-start-sm-1{grid-column-start:1}.panel-grid .g-start-sm-2{grid-column-start:2}.panel-grid .g-start-sm-3{grid-column-start:3}.panel-grid .g-start-sm-4{grid-column-start:4}.panel-grid .g-start-sm-5{grid-column-start:5}.panel-grid .g-start-sm-6{grid-column-start:6}.panel-grid .g-start-sm-7{grid-column-start:7}.panel-grid .g-start-sm-8{grid-column-start:8}.panel-grid .g-start-sm-9{grid-column-start:9}.panel-grid .g-start-sm-10{grid-column-start:10}.panel-grid .g-start-sm-11{grid-column-start:11}.panel-grid .g-start-sm-12{grid-column-start:12}.panel-grid .g-start-sm-13{grid-column-start:13}.panel-grid .g-start-sm-14{grid-column-start:14}.panel-grid .g-start-sm-15{grid-column-start:15}.panel-grid .g-start-sm-16{grid-column-start:16}.panel-grid .g-start-sm-17{grid-column-start:17}.panel-grid .g-start-sm-18{grid-column-start:18}.panel-grid .g-start-sm-19{grid-column-start:19}.panel-grid .g-start-sm-20{grid-column-start:20}.panel-grid .g-start-sm-21{grid-column-start:21}.panel-grid .g-start-sm-22{grid-column-start:22}.panel-grid .g-start-sm-23{grid-column-start:23}}@media(min-width: 768px){.panel-grid .g-col-md-1{grid-column:auto/span 1}.panel-grid .g-col-md-2{grid-column:auto/span 2}.panel-grid .g-col-md-3{grid-column:auto/span 3}.panel-grid .g-col-md-4{grid-column:auto/span 4}.panel-grid .g-col-md-5{grid-column:auto/span 5}.panel-grid .g-col-md-6{grid-column:auto/span 6}.panel-grid .g-col-md-7{grid-column:auto/span 7}.panel-grid .g-col-md-8{grid-column:auto/span 8}.panel-grid .g-col-md-9{grid-column:auto/span 9}.panel-grid .g-col-md-10{grid-column:auto/span 10}.panel-grid .g-col-md-11{grid-column:auto/span 11}.panel-grid .g-col-md-12{grid-column:auto/span 12}.panel-grid .g-col-md-13{grid-column:auto/span 13}.panel-grid .g-col-md-14{grid-column:auto/span 14}.panel-grid .g-col-md-15{grid-column:auto/span 15}.panel-grid .g-col-md-16{grid-column:auto/span 16}.panel-grid .g-col-md-17{grid-column:auto/span 17}.panel-grid .g-col-md-18{grid-column:auto/span 18}.panel-grid .g-col-md-19{grid-column:auto/span 19}.panel-grid .g-col-md-20{grid-column:auto/span 20}.panel-grid .g-col-md-21{grid-column:auto/span 21}.panel-grid .g-col-md-22{grid-column:auto/span 22}.panel-grid .g-col-md-23{grid-column:auto/span 23}.panel-grid .g-col-md-24{grid-column:auto/span 24}.panel-grid .g-start-md-1{grid-column-start:1}.panel-grid .g-start-md-2{grid-column-start:2}.panel-grid .g-start-md-3{grid-column-start:3}.panel-grid .g-start-md-4{grid-column-start:4}.panel-grid .g-start-md-5{grid-column-start:5}.panel-grid .g-start-md-6{grid-column-start:6}.panel-grid .g-start-md-7{grid-column-start:7}.panel-grid .g-start-md-8{grid-column-start:8}.panel-grid .g-start-md-9{grid-column-start:9}.panel-grid .g-start-md-10{grid-column-start:10}.panel-grid .g-start-md-11{grid-column-start:11}.panel-grid .g-start-md-12{grid-column-start:12}.panel-grid .g-start-md-13{grid-column-start:13}.panel-grid .g-start-md-14{grid-column-start:14}.panel-grid .g-start-md-15{grid-column-start:15}.panel-grid .g-start-md-16{grid-column-start:16}.panel-grid .g-start-md-17{grid-column-start:17}.panel-grid .g-start-md-18{grid-column-start:18}.panel-grid .g-start-md-19{grid-column-start:19}.panel-grid .g-start-md-20{grid-column-start:20}.panel-grid .g-start-md-21{grid-column-start:21}.panel-grid .g-start-md-22{grid-column-start:22}.panel-grid .g-start-md-23{grid-column-start:23}}@media(min-width: 992px){.panel-grid .g-col-lg-1{grid-column:auto/span 1}.panel-grid .g-col-lg-2{grid-column:auto/span 2}.panel-grid .g-col-lg-3{grid-column:auto/span 3}.panel-grid .g-col-lg-4{grid-column:auto/span 4}.panel-grid .g-col-lg-5{grid-column:auto/span 5}.panel-grid .g-col-lg-6{grid-column:auto/span 6}.panel-grid .g-col-lg-7{grid-column:auto/span 7}.panel-grid .g-col-lg-8{grid-column:auto/span 8}.panel-grid .g-col-lg-9{grid-column:auto/span 9}.panel-grid .g-col-lg-10{grid-column:auto/span 10}.panel-grid .g-col-lg-11{grid-column:auto/span 11}.panel-grid .g-col-lg-12{grid-column:auto/span 12}.panel-grid .g-col-lg-13{grid-column:auto/span 13}.panel-grid .g-col-lg-14{grid-column:auto/span 14}.panel-grid .g-col-lg-15{grid-column:auto/span 15}.panel-grid .g-col-lg-16{grid-column:auto/span 16}.panel-grid .g-col-lg-17{grid-column:auto/span 17}.panel-grid .g-col-lg-18{grid-column:auto/span 18}.panel-grid .g-col-lg-19{grid-column:auto/span 19}.panel-grid .g-col-lg-20{grid-column:auto/span 20}.panel-grid .g-col-lg-21{grid-column:auto/span 21}.panel-grid .g-col-lg-22{grid-column:auto/span 22}.panel-grid .g-col-lg-23{grid-column:auto/span 23}.panel-grid .g-col-lg-24{grid-column:auto/span 24}.panel-grid .g-start-lg-1{grid-column-start:1}.panel-grid .g-start-lg-2{grid-column-start:2}.panel-grid .g-start-lg-3{grid-column-start:3}.panel-grid .g-start-lg-4{grid-column-start:4}.panel-grid .g-start-lg-5{grid-column-start:5}.panel-grid .g-start-lg-6{grid-column-start:6}.panel-grid .g-start-lg-7{grid-column-start:7}.panel-grid .g-start-lg-8{grid-column-start:8}.panel-grid .g-start-lg-9{grid-column-start:9}.panel-grid .g-start-lg-10{grid-column-start:10}.panel-grid .g-start-lg-11{grid-column-start:11}.panel-grid .g-start-lg-12{grid-column-start:12}.panel-grid .g-start-lg-13{grid-column-start:13}.panel-grid .g-start-lg-14{grid-column-start:14}.panel-grid .g-start-lg-15{grid-column-start:15}.panel-grid .g-start-lg-16{grid-column-start:16}.panel-grid .g-start-lg-17{grid-column-start:17}.panel-grid .g-start-lg-18{grid-column-start:18}.panel-grid .g-start-lg-19{grid-column-start:19}.panel-grid .g-start-lg-20{grid-column-start:20}.panel-grid .g-start-lg-21{grid-column-start:21}.panel-grid .g-start-lg-22{grid-column-start:22}.panel-grid .g-start-lg-23{grid-column-start:23}}@media(min-width: 1200px){.panel-grid .g-col-xl-1{grid-column:auto/span 1}.panel-grid .g-col-xl-2{grid-column:auto/span 2}.panel-grid .g-col-xl-3{grid-column:auto/span 3}.panel-grid .g-col-xl-4{grid-column:auto/span 4}.panel-grid .g-col-xl-5{grid-column:auto/span 5}.panel-grid .g-col-xl-6{grid-column:auto/span 6}.panel-grid .g-col-xl-7{grid-column:auto/span 7}.panel-grid .g-col-xl-8{grid-column:auto/span 8}.panel-grid .g-col-xl-9{grid-column:auto/span 9}.panel-grid .g-col-xl-10{grid-column:auto/span 10}.panel-grid .g-col-xl-11{grid-column:auto/span 11}.panel-grid .g-col-xl-12{grid-column:auto/span 12}.panel-grid .g-col-xl-13{grid-column:auto/span 13}.panel-grid .g-col-xl-14{grid-column:auto/span 14}.panel-grid .g-col-xl-15{grid-column:auto/span 15}.panel-grid .g-col-xl-16{grid-column:auto/span 16}.panel-grid .g-col-xl-17{grid-column:auto/span 17}.panel-grid .g-col-xl-18{grid-column:auto/span 18}.panel-grid .g-col-xl-19{grid-column:auto/span 19}.panel-grid .g-col-xl-20{grid-column:auto/span 20}.panel-grid .g-col-xl-21{grid-column:auto/span 21}.panel-grid .g-col-xl-22{grid-column:auto/span 22}.panel-grid .g-col-xl-23{grid-column:auto/span 23}.panel-grid .g-col-xl-24{grid-column:auto/span 24}.panel-grid .g-start-xl-1{grid-column-start:1}.panel-grid .g-start-xl-2{grid-column-start:2}.panel-grid .g-start-xl-3{grid-column-start:3}.panel-grid .g-start-xl-4{grid-column-start:4}.panel-grid .g-start-xl-5{grid-column-start:5}.panel-grid .g-start-xl-6{grid-column-start:6}.panel-grid .g-start-xl-7{grid-column-start:7}.panel-grid .g-start-xl-8{grid-column-start:8}.panel-grid .g-start-xl-9{grid-column-start:9}.panel-grid .g-start-xl-10{grid-column-start:10}.panel-grid .g-start-xl-11{grid-column-start:11}.panel-grid .g-start-xl-12{grid-column-start:12}.panel-grid .g-start-xl-13{grid-column-start:13}.panel-grid .g-start-xl-14{grid-column-start:14}.panel-grid .g-start-xl-15{grid-column-start:15}.panel-grid .g-start-xl-16{grid-column-start:16}.panel-grid .g-start-xl-17{grid-column-start:17}.panel-grid .g-start-xl-18{grid-column-start:18}.panel-grid .g-start-xl-19{grid-column-start:19}.panel-grid .g-start-xl-20{grid-column-start:20}.panel-grid .g-start-xl-21{grid-column-start:21}.panel-grid .g-start-xl-22{grid-column-start:22}.panel-grid .g-start-xl-23{grid-column-start:23}}@media(min-width: 1400px){.panel-grid .g-col-xxl-1{grid-column:auto/span 1}.panel-grid .g-col-xxl-2{grid-column:auto/span 2}.panel-grid .g-col-xxl-3{grid-column:auto/span 3}.panel-grid .g-col-xxl-4{grid-column:auto/span 4}.panel-grid .g-col-xxl-5{grid-column:auto/span 5}.panel-grid .g-col-xxl-6{grid-column:auto/span 6}.panel-grid .g-col-xxl-7{grid-column:auto/span 7}.panel-grid .g-col-xxl-8{grid-column:auto/span 8}.panel-grid .g-col-xxl-9{grid-column:auto/span 9}.panel-grid .g-col-xxl-10{grid-column:auto/span 10}.panel-grid .g-col-xxl-11{grid-column:auto/span 11}.panel-grid .g-col-xxl-12{grid-column:auto/span 12}.panel-grid .g-col-xxl-13{grid-column:auto/span 13}.panel-grid .g-col-xxl-14{grid-column:auto/span 14}.panel-grid .g-col-xxl-15{grid-column:auto/span 15}.panel-grid .g-col-xxl-16{grid-column:auto/span 16}.panel-grid .g-col-xxl-17{grid-column:auto/span 17}.panel-grid .g-col-xxl-18{grid-column:auto/span 18}.panel-grid .g-col-xxl-19{grid-column:auto/span 19}.panel-grid .g-col-xxl-20{grid-column:auto/span 20}.panel-grid .g-col-xxl-21{grid-column:auto/span 21}.panel-grid .g-col-xxl-22{grid-column:auto/span 22}.panel-grid .g-col-xxl-23{grid-column:auto/span 23}.panel-grid .g-col-xxl-24{grid-column:auto/span 24}.panel-grid .g-start-xxl-1{grid-column-start:1}.panel-grid .g-start-xxl-2{grid-column-start:2}.panel-grid .g-start-xxl-3{grid-column-start:3}.panel-grid .g-start-xxl-4{grid-column-start:4}.panel-grid .g-start-xxl-5{grid-column-start:5}.panel-grid .g-start-xxl-6{grid-column-start:6}.panel-grid .g-start-xxl-7{grid-column-start:7}.panel-grid .g-start-xxl-8{grid-column-start:8}.panel-grid .g-start-xxl-9{grid-column-start:9}.panel-grid .g-start-xxl-10{grid-column-start:10}.panel-grid .g-start-xxl-11{grid-column-start:11}.panel-grid .g-start-xxl-12{grid-column-start:12}.panel-grid .g-start-xxl-13{grid-column-start:13}.panel-grid .g-start-xxl-14{grid-column-start:14}.panel-grid .g-start-xxl-15{grid-column-start:15}.panel-grid .g-start-xxl-16{grid-column-start:16}.panel-grid .g-start-xxl-17{grid-column-start:17}.panel-grid .g-start-xxl-18{grid-column-start:18}.panel-grid .g-start-xxl-19{grid-column-start:19}.panel-grid .g-start-xxl-20{grid-column-start:20}.panel-grid .g-start-xxl-21{grid-column-start:21}.panel-grid .g-start-xxl-22{grid-column-start:22}.panel-grid .g-start-xxl-23{grid-column-start:23}}main{margin-top:1em;margin-bottom:1em}h1,.h1,h2,.h2{opacity:.9;margin-top:2rem;margin-bottom:1rem;font-weight:600}h1.title,.title.h1{margin-top:0}h2,.h2{border-bottom:1px solid #dee2e6;padding-bottom:.5rem}h3,.h3{font-weight:600}h3,.h3,h4,.h4{opacity:.9;margin-top:1.5rem}h5,.h5,h6,.h6{opacity:.9}.header-section-number{color:#737373}.nav-link.active .header-section-number{color:inherit}mark,.mark{padding:0em}.panel-caption,caption,.figure-caption{font-size:.9rem}.panel-caption,.figure-caption,figcaption{color:#737373}.table-caption,caption{color:#333}.quarto-layout-cell[data-ref-parent] caption{color:#737373}.column-margin figcaption,.margin-caption,div.aside,aside,.column-margin{color:#737373;font-size:.825rem}.panel-caption.margin-caption{text-align:inherit}.column-margin.column-container p{margin-bottom:0}.column-margin.column-container>*:not(.collapse){padding-top:.5em;padding-bottom:.5em;display:block}.column-margin.column-container>*.collapse:not(.show){display:none}@media(min-width: 768px){.column-margin.column-container .callout-margin-content:first-child{margin-top:4.5em}.column-margin.column-container .callout-margin-content-simple:first-child{margin-top:3.5em}}.margin-caption>*{padding-top:.5em;padding-bottom:.5em}@media(max-width: 767.98px){.quarto-layout-row{flex-direction:column}}.nav-tabs .nav-item{margin-top:1px;cursor:pointer}.tab-content{margin-top:0px;border-left:#dee2e6 1px solid;border-right:#dee2e6 1px solid;border-bottom:#dee2e6 1px solid;margin-left:0;padding:1em;margin-bottom:1em}@media(max-width: 767.98px){.layout-sidebar{margin-left:0;margin-right:0}}.panel-sidebar,.panel-sidebar .form-control,.panel-input,.panel-input .form-control,.selectize-dropdown{font-size:.9rem}.panel-sidebar .form-control,.panel-input .form-control{padding-top:.1rem}.tab-pane div.sourceCode{margin-top:0px}.tab-pane>p{padding-top:1em}.tab-content>.tab-pane:not(.active){display:none !important}div.sourceCode{background-color:rgba(233,236,239,.65);border:1px solid rgba(233,236,239,.65);border-radius:.25rem}pre.sourceCode{background-color:rgba(0,0,0,0)}pre.sourceCode{border:none;font-size:.875em;overflow:visible !important;padding:.4em}.callout pre.sourceCode{padding-left:0}div.sourceCode{overflow-y:hidden}.callout div.sourceCode{margin-left:initial}.blockquote{font-size:inherit;padding-left:1rem;padding-right:1.5rem;color:#737373}.blockquote h1:first-child,.blockquote .h1:first-child,.blockquote h2:first-child,.blockquote .h2:first-child,.blockquote h3:first-child,.blockquote .h3:first-child,.blockquote h4:first-child,.blockquote .h4:first-child,.blockquote h5:first-child,.blockquote .h5:first-child{margin-top:0}pre{background-color:initial;padding:initial;border:initial}p code:not(.sourceCode),li code:not(.sourceCode),td code:not(.sourceCode){background-color:#f7f7f7;padding:.2em}nav p code:not(.sourceCode),nav li code:not(.sourceCode),nav td code:not(.sourceCode){background-color:rgba(0,0,0,0);padding:0}td code:not(.sourceCode){white-space:pre-wrap}#quarto-embedded-source-code-modal>.modal-dialog{max-width:1000px;padding-left:1.75rem;padding-right:1.75rem}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-body{padding:0}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-body div.sourceCode{margin:0;padding:.2rem .2rem;border-radius:0px;border:none}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-header{padding:.7rem}.code-tools-button{font-size:1rem;padding:.15rem .15rem;margin-left:5px;color:#6c757d;background-color:rgba(0,0,0,0);transition:initial;cursor:pointer}.code-tools-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}.code-tools-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}#quarto-embedded-source-code-modal .code-copy-button>.bi::before{background-image:url('data:image/svg+xml,')}#quarto-embedded-source-code-modal .code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}.sidebar{will-change:top;transition:top 200ms linear;position:sticky;overflow-y:auto;padding-top:1.2em;max-height:100vh}.sidebar.toc-left,.sidebar.margin-sidebar{top:0px;padding-top:1em}.sidebar.toc-left>*,.sidebar.margin-sidebar>*{padding-top:.5em}.sidebar.quarto-banner-title-block-sidebar>*{padding-top:1.65em}figure .quarto-notebook-link{margin-top:.5em}.quarto-notebook-link{font-size:.75em;color:#6c757d;margin-bottom:1em;text-decoration:none;display:block}.quarto-notebook-link:hover{text-decoration:underline;color:#e95420}.quarto-notebook-link::before{display:inline-block;height:.75rem;width:.75rem;margin-bottom:0em;margin-right:.25em;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:.75rem .75rem}.quarto-alternate-notebooks i.bi,.quarto-alternate-formats i.bi{margin-right:.4em}.quarto-notebook .cell-container{display:flex}.quarto-notebook .cell-container .cell{flex-grow:4}.quarto-notebook .cell-container .cell-decorator{padding-top:1.5em;padding-right:1em;text-align:right}.quarto-notebook h2,.quarto-notebook .h2{border-bottom:none}.sidebar .quarto-alternate-formats a,.sidebar .quarto-alternate-notebooks a{text-decoration:none}.sidebar .quarto-alternate-formats a:hover,.sidebar .quarto-alternate-notebooks a:hover{color:#e95420}.sidebar .quarto-alternate-notebooks h2,.sidebar .quarto-alternate-notebooks .h2,.sidebar .quarto-alternate-formats h2,.sidebar .quarto-alternate-formats .h2,.sidebar nav[role=doc-toc]>h2,.sidebar nav[role=doc-toc]>.h2{font-size:.875rem;font-weight:400;margin-bottom:.5rem;margin-top:.3rem;font-family:inherit;border-bottom:0;padding-bottom:0;padding-top:0px}.sidebar .quarto-alternate-notebooks h2,.sidebar .quarto-alternate-notebooks .h2,.sidebar .quarto-alternate-formats h2,.sidebar .quarto-alternate-formats .h2{margin-top:1rem}.sidebar nav[role=doc-toc]>ul a{border-left:1px solid #e9ecef;padding-left:.6rem}.sidebar .quarto-alternate-notebooks h2>ul a,.sidebar .quarto-alternate-notebooks .h2>ul a,.sidebar .quarto-alternate-formats h2>ul a,.sidebar .quarto-alternate-formats .h2>ul a{border-left:none;padding-left:.6rem}.sidebar .quarto-alternate-notebooks ul a:empty,.sidebar .quarto-alternate-formats ul a:empty,.sidebar nav[role=doc-toc]>ul a:empty{display:none}.sidebar .quarto-alternate-notebooks ul,.sidebar .quarto-alternate-formats ul,.sidebar nav[role=doc-toc] ul{padding-left:0;list-style:none;font-size:.875rem;font-weight:300}.sidebar .quarto-alternate-notebooks ul li a,.sidebar .quarto-alternate-formats ul li a,.sidebar nav[role=doc-toc]>ul li a{line-height:1.1rem;padding-bottom:.2rem;padding-top:.2rem;color:inherit}.sidebar nav[role=doc-toc] ul>li>ul>li>a{padding-left:1.2em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>a{padding-left:2.4em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>a{padding-left:3.6em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>ul>li>a{padding-left:4.8em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>ul>li>ul>li>a{padding-left:6em}.sidebar nav[role=doc-toc] ul>li>a.active,.sidebar nav[role=doc-toc] ul>li>ul>li>a.active{border-left:1px solid #e95420;color:#e95420 !important}.sidebar nav[role=doc-toc] ul>li>a:hover,.sidebar nav[role=doc-toc] ul>li>ul>li>a:hover{color:#e95420 !important}kbd,.kbd{color:#333;background-color:#f8f9fa;border:1px solid;border-radius:5px;border-color:#dee2e6}div.hanging-indent{margin-left:1em;text-indent:-1em}.citation a,.footnote-ref{text-decoration:none}.footnotes ol{padding-left:1em}.tippy-content>*{margin-bottom:.7em}.tippy-content>*:last-child{margin-bottom:0}.table a{word-break:break-word}.table>thead{border-top-width:1px;border-top-color:#dee2e6;border-bottom:1px solid #b3b3b3}.callout{margin-top:1.25rem;margin-bottom:1.25rem;border-radius:.25rem;overflow-wrap:break-word}.callout .callout-title-container{overflow-wrap:anywhere}.callout.callout-style-simple{padding:.4em .7em;border-left:5px solid;border-right:1px solid #dee2e6;border-top:1px solid #dee2e6;border-bottom:1px solid #dee2e6}.callout.callout-style-default{border-left:5px solid;border-right:1px solid #dee2e6;border-top:1px solid #dee2e6;border-bottom:1px solid #dee2e6}.callout .callout-body-container{flex-grow:1}.callout.callout-style-simple .callout-body{font-size:.9rem;font-weight:400}.callout.callout-style-default .callout-body{font-size:.9rem;font-weight:400}.callout.callout-titled .callout-body{margin-top:.2em}.callout:not(.no-icon).callout-titled.callout-style-simple .callout-body{padding-left:1.6em}.callout.callout-titled>.callout-header{padding-top:.2em;margin-bottom:-0.2em}.callout.callout-style-simple>div.callout-header{border-bottom:none;font-size:.9rem;font-weight:600;opacity:75%}.callout.callout-style-default>div.callout-header{border-bottom:none;font-weight:600;opacity:85%;font-size:.9rem;padding-left:.5em;padding-right:.5em}.callout.callout-style-default div.callout-body{padding-left:.5em;padding-right:.5em}.callout.callout-style-default div.callout-body>:first-child{margin-top:.5em}.callout>div.callout-header[data-bs-toggle=collapse]{cursor:pointer}.callout.callout-style-default .callout-header[aria-expanded=false],.callout.callout-style-default .callout-header[aria-expanded=true]{padding-top:0px;margin-bottom:0px;align-items:center}.callout.callout-titled .callout-body>:last-child:not(.sourceCode),.callout.callout-titled .callout-body>div>:last-child:not(.sourceCode){margin-bottom:.5rem}.callout:not(.callout-titled) .callout-body>:first-child,.callout:not(.callout-titled) .callout-body>div>:first-child{margin-top:.25rem}.callout:not(.callout-titled) .callout-body>:last-child,.callout:not(.callout-titled) .callout-body>div>:last-child{margin-bottom:.2rem}.callout.callout-style-simple .callout-icon::before,.callout.callout-style-simple .callout-toggle::before{height:1rem;width:1rem;display:inline-block;content:"";background-repeat:no-repeat;background-size:1rem 1rem}.callout.callout-style-default .callout-icon::before,.callout.callout-style-default .callout-toggle::before{height:.9rem;width:.9rem;display:inline-block;content:"";background-repeat:no-repeat;background-size:.9rem .9rem}.callout.callout-style-default .callout-toggle::before{margin-top:5px}.callout .callout-btn-toggle .callout-toggle::before{transition:transform .2s linear}.callout .callout-header[aria-expanded=false] .callout-toggle::before{transform:rotate(-90deg)}.callout .callout-header[aria-expanded=true] .callout-toggle::before{transform:none}.callout.callout-style-simple:not(.no-icon) div.callout-icon-container{padding-top:.2em;padding-right:.55em}.callout.callout-style-default:not(.no-icon) div.callout-icon-container{padding-top:.1em;padding-right:.35em}.callout.callout-style-default:not(.no-icon) div.callout-title-container{margin-top:-1px}.callout.callout-style-default.callout-caution:not(.no-icon) div.callout-icon-container{padding-top:.3em;padding-right:.35em}.callout>.callout-body>.callout-icon-container>.no-icon,.callout>.callout-header>.callout-icon-container>.no-icon{display:none}div.callout.callout{border-left-color:#6c757d}div.callout.callout-style-default>.callout-header{background-color:#6c757d}div.callout-note.callout{border-left-color:#007bff}div.callout-note.callout-style-default>.callout-header{background-color:#e6f2ff}div.callout-note:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-note.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-note .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-tip.callout{border-left-color:#38b44a}div.callout-tip.callout-style-default>.callout-header{background-color:#ebf8ed}div.callout-tip:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-tip.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-tip .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-warning.callout{border-left-color:#efb73e}div.callout-warning.callout-style-default>.callout-header{background-color:#fdf8ec}div.callout-warning:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-warning.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-warning .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-caution.callout{border-left-color:#e95420}div.callout-caution.callout-style-default>.callout-header{background-color:#fdeee9}div.callout-caution:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-caution.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-caution .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-important.callout{border-left-color:#df382c}div.callout-important.callout-style-default>.callout-header{background-color:#fcebea}div.callout-important:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-important.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-important .callout-toggle::before{background-image:url('data:image/svg+xml,')}.quarto-toggle-container{display:flex;align-items:center}.quarto-reader-toggle .bi::before,.quarto-color-scheme-toggle .bi::before{display:inline-block;height:1rem;width:1rem;content:"";background-repeat:no-repeat;background-size:1rem 1rem}.sidebar-navigation{padding-left:20px}.navbar .quarto-color-scheme-toggle:not(.alternate) .bi::before{background-image:url('data:image/svg+xml,')}.navbar .quarto-color-scheme-toggle.alternate .bi::before{background-image:url('data:image/svg+xml,')}.sidebar-navigation .quarto-color-scheme-toggle:not(.alternate) .bi::before{background-image:url('data:image/svg+xml,')}.sidebar-navigation .quarto-color-scheme-toggle.alternate .bi::before{background-image:url('data:image/svg+xml,')}.quarto-sidebar-toggle{border-color:#dee2e6;border-bottom-left-radius:.25rem;border-bottom-right-radius:.25rem;border-style:solid;border-width:1px;overflow:hidden;border-top-width:0px;padding-top:0px !important}.quarto-sidebar-toggle-title{cursor:pointer;padding-bottom:2px;margin-left:.25em;text-align:center;font-weight:400;font-size:.775em}#quarto-content .quarto-sidebar-toggle{background:#fafafa}#quarto-content .quarto-sidebar-toggle-title{color:#333}.quarto-sidebar-toggle-icon{color:#dee2e6;margin-right:.5em;float:right;transition:transform .2s ease}.quarto-sidebar-toggle-icon::before{padding-top:5px}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-icon{transform:rotate(-180deg)}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-title{border-bottom:solid #dee2e6 1px}.quarto-sidebar-toggle-contents{background-color:#fff;padding-right:10px;padding-left:10px;margin-top:0px !important;transition:max-height .5s ease}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-contents{padding-top:1em;padding-bottom:10px}.quarto-sidebar-toggle:not(.expanded) .quarto-sidebar-toggle-contents{padding-top:0px !important;padding-bottom:0px}nav[role=doc-toc]{z-index:1020}#quarto-sidebar>*,nav[role=doc-toc]>*{transition:opacity .1s ease,border .1s ease}#quarto-sidebar.slow>*,nav[role=doc-toc].slow>*{transition:opacity .4s ease,border .4s ease}.quarto-color-scheme-toggle:not(.alternate).top-right .bi::before{background-image:url('data:image/svg+xml,')}.quarto-color-scheme-toggle.alternate.top-right .bi::before{background-image:url('data:image/svg+xml,')}#quarto-appendix.default{border-top:1px solid #dee2e6}#quarto-appendix.default{background-color:#fff;padding-top:1.5em;margin-top:2em;z-index:998}#quarto-appendix.default .quarto-appendix-heading{margin-top:0;line-height:1.4em;font-weight:600;opacity:.9;border-bottom:none;margin-bottom:0}#quarto-appendix.default .footnotes ol,#quarto-appendix.default .footnotes ol li>p:last-of-type,#quarto-appendix.default .quarto-appendix-contents>p:last-of-type{margin-bottom:0}#quarto-appendix.default .quarto-appendix-secondary-label{margin-bottom:.4em}#quarto-appendix.default .quarto-appendix-bibtex{font-size:.7em;padding:1em;border:solid 1px #dee2e6;margin-bottom:1em}#quarto-appendix.default .quarto-appendix-bibtex code.sourceCode{white-space:pre-wrap}#quarto-appendix.default .quarto-appendix-citeas{font-size:.9em;padding:1em;border:solid 1px #dee2e6;margin-bottom:1em}#quarto-appendix.default .quarto-appendix-heading{font-size:1em !important}#quarto-appendix.default *[role=doc-endnotes]>ol,#quarto-appendix.default .quarto-appendix-contents>*:not(h2):not(.h2){font-size:.9em}#quarto-appendix.default section{padding-bottom:1.5em}#quarto-appendix.default section *[role=doc-endnotes],#quarto-appendix.default section>*:not(a){opacity:.9;word-wrap:break-word}.btn.btn-quarto,div.cell-output-display .btn-quarto{color:#26282a;background-color:#adb5bd;border-color:#adb5bd}.btn.btn-quarto:hover,div.cell-output-display .btn-quarto:hover{color:#26282a;background-color:#b9c0c7;border-color:#b5bcc4}.btn-check:focus+.btn.btn-quarto,.btn.btn-quarto:focus,.btn-check:focus+div.cell-output-display .btn-quarto,div.cell-output-display .btn-quarto:focus{color:#26282a;background-color:#b9c0c7;border-color:#b5bcc4;box-shadow:0 0 0 .25rem rgba(153,160,167,.5)}.btn-check:checked+.btn.btn-quarto,.btn-check:active+.btn.btn-quarto,.btn.btn-quarto:active,.btn.btn-quarto.active,.show>.btn.btn-quarto.dropdown-toggle,.btn-check:checked+div.cell-output-display .btn-quarto,.btn-check:active+div.cell-output-display .btn-quarto,div.cell-output-display .btn-quarto:active,div.cell-output-display .btn-quarto.active,.show>div.cell-output-display .btn-quarto.dropdown-toggle{color:#000;background-color:#bdc4ca;border-color:#b5bcc4}.btn-check:checked+.btn.btn-quarto:focus,.btn-check:active+.btn.btn-quarto:focus,.btn.btn-quarto:active:focus,.btn.btn-quarto.active:focus,.show>.btn.btn-quarto.dropdown-toggle:focus,.btn-check:checked+div.cell-output-display .btn-quarto:focus,.btn-check:active+div.cell-output-display .btn-quarto:focus,div.cell-output-display .btn-quarto:active:focus,div.cell-output-display .btn-quarto.active:focus,.show>div.cell-output-display .btn-quarto.dropdown-toggle:focus{box-shadow:0 0 0 .25rem rgba(153,160,167,.5)}.btn.btn-quarto:disabled,.btn.btn-quarto.disabled,div.cell-output-display .btn-quarto:disabled,div.cell-output-display .btn-quarto.disabled{color:#fff;background-color:#adb5bd;border-color:#adb5bd}nav.quarto-secondary-nav.color-navbar{background-color:#e95420;color:rgba(255,255,255,.8)}nav.quarto-secondary-nav.color-navbar h1,nav.quarto-secondary-nav.color-navbar .h1,nav.quarto-secondary-nav.color-navbar .quarto-btn-toggle{color:rgba(255,255,255,.8)}@media(max-width: 991.98px){body.nav-sidebar .quarto-title-banner{margin-bottom:0;padding-bottom:0}body.nav-sidebar #title-block-header{margin-block-end:0}}p.subtitle{margin-top:.25em;margin-bottom:.5em}code a:any-link{color:inherit;text-decoration-color:#6c757d}/*! light */div.observablehq table thead tr th{background-color:var(--bs-body-bg)}input,button,select,optgroup,textarea{background-color:var(--bs-body-bg)}.code-annotated .code-copy-button{margin-right:1.25em;margin-top:0;padding-bottom:0;padding-top:3px}.code-annotation-gutter-bg{background-color:#fff}.code-annotation-gutter{background-color:rgba(233,236,239,.65)}.code-annotation-gutter,.code-annotation-gutter-bg{height:100%;width:calc(20px + .5em);position:absolute;top:0;right:0}dl.code-annotation-container-grid dt{margin-right:1em;margin-top:.25rem}dl.code-annotation-container-grid dt{font-family:var(--bs-font-monospace);color:#4d4d4d;border:solid #4d4d4d 1px;border-radius:50%;height:22px;width:22px;line-height:22px;font-size:11px;text-align:center;vertical-align:middle;text-decoration:none}dl.code-annotation-container-grid dt[data-target-cell]{cursor:pointer}dl.code-annotation-container-grid dt[data-target-cell].code-annotation-active{color:#fff;border:solid #aaa 1px;background-color:#aaa}pre.code-annotation-code{padding-top:0;padding-bottom:0}pre.code-annotation-code code{z-index:3}#code-annotation-line-highlight-gutter{width:100%;border-top:solid rgba(170,170,170,.2666666667) 1px;border-bottom:solid rgba(170,170,170,.2666666667) 1px;z-index:2;background-color:rgba(170,170,170,.1333333333)}#code-annotation-line-highlight{margin-left:-4em;width:calc(100% + 4em);border-top:solid rgba(170,170,170,.2666666667) 1px;border-bottom:solid rgba(170,170,170,.2666666667) 1px;z-index:2;background-color:rgba(170,170,170,.1333333333)}code.sourceCode .code-annotation-anchor.code-annotation-active{background-color:var(--quarto-hl-normal-color, #aaaaaa);border:solid var(--quarto-hl-normal-color, #aaaaaa) 1px;color:#e9ecef;font-weight:bolder}code.sourceCode .code-annotation-anchor{font-family:var(--bs-font-monospace);color:var(--quarto-hl-co-color);border:solid var(--quarto-hl-co-color) 1px;border-radius:50%;height:18px;width:18px;font-size:9px;margin-top:2px}code.sourceCode button.code-annotation-anchor{padding:2px}code.sourceCode a.code-annotation-anchor{line-height:18px;text-align:center;vertical-align:middle;cursor:default;text-decoration:none}@media print{.page-columns .column-screen-inset{grid-column:page-start-inset/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset table{background:#fff}.page-columns .column-screen-inset-left{grid-column:page-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-left table{background:#fff}.page-columns .column-screen-inset-right{grid-column:body-content-start/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-right table{background:#fff}.page-columns .column-screen{grid-column:page-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen table{background:#fff}.page-columns .column-screen-left{grid-column:page-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-left table{background:#fff}.page-columns .column-screen-right{grid-column:body-content-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-right table{background:#fff}.page-columns .column-screen-inset-shaded{grid-column:page-start-inset/page-end-inset;padding:1em;background:#e9ecef;z-index:998;transform:translate3d(0, 0, 0);margin-bottom:1em}}.quarto-video{margin-bottom:1em}.table>thead{border-top-width:0}.table>:not(caption)>*:not(:last-child)>*{border-bottom-color:#e6e6e6;border-bottom-style:solid;border-bottom-width:1px}.table>:not(:first-child){border-top:1px solid #b3b3b3;border-bottom:1px solid inherit}.table tbody{border-bottom-color:#b3b3b3}a.external:after{display:inline-block;height:.75rem;width:.75rem;margin-bottom:.15em;margin-left:.25em;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:.75rem .75rem}div.sourceCode code a.external:after{content:none}a.external:after:hover{cursor:pointer}.quarto-ext-icon{display:inline-block;font-size:.75em;padding-left:.3em}.code-with-filename .code-with-filename-file{margin-bottom:0;padding-bottom:2px;padding-top:2px;padding-left:.7em;border:var(--quarto-border-width) solid var(--quarto-border-color);border-radius:var(--quarto-border-radius);border-bottom:0;border-bottom-left-radius:0%;border-bottom-right-radius:0%}.code-with-filename div.sourceCode,.reveal .code-with-filename div.sourceCode{margin-top:0;border-top-left-radius:0%;border-top-right-radius:0%}.code-with-filename .code-with-filename-file pre{margin-bottom:0}.code-with-filename .code-with-filename-file,.code-with-filename .code-with-filename-file pre{background-color:rgba(219,219,219,.8)}.quarto-dark .code-with-filename .code-with-filename-file,.quarto-dark .code-with-filename .code-with-filename-file pre{background-color:#555}.code-with-filename .code-with-filename-file strong{font-weight:400}.badge.bg-light{color:#333}.quarto-title-banner{margin-bottom:1em;color:rgba(255,255,255,.8);background:#e95420}.quarto-title-banner .code-tools-button{color:rgba(204,204,204,.8)}.quarto-title-banner .code-tools-button:hover{color:rgba(255,255,255,.8)}.quarto-title-banner .code-tools-button>.bi::before{background-image:url('data:image/svg+xml,')}.quarto-title-banner .code-tools-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}.quarto-title-banner .quarto-title .title{font-weight:600}.quarto-title-banner .quarto-categories{margin-top:.75em}@media(min-width: 992px){.quarto-title-banner{padding-top:2.5em;padding-bottom:2.5em}}@media(max-width: 991.98px){.quarto-title-banner{padding-top:1em;padding-bottom:1em}}main.quarto-banner-title-block>section:first-child>h2,main.quarto-banner-title-block>section:first-child>.h2,main.quarto-banner-title-block>section:first-child>h3,main.quarto-banner-title-block>section:first-child>.h3,main.quarto-banner-title-block>section:first-child>h4,main.quarto-banner-title-block>section:first-child>.h4{margin-top:0}.quarto-title .quarto-categories{display:flex;flex-wrap:wrap;row-gap:.5em;column-gap:.4em;padding-bottom:.5em;margin-top:.75em}.quarto-title .quarto-categories .quarto-category{padding:.25em .75em;font-size:.65em;text-transform:uppercase;border:solid 1px;border-radius:.25rem;opacity:.6}.quarto-title .quarto-categories .quarto-category a{color:inherit}#title-block-header.quarto-title-block.default .quarto-title-meta{display:grid;grid-template-columns:repeat(2, 1fr)}#title-block-header.quarto-title-block.default .quarto-title .title{margin-bottom:0}#title-block-header.quarto-title-block.default .quarto-title-author-orcid img{margin-top:-5px}#title-block-header.quarto-title-block.default .quarto-description p:last-of-type{margin-bottom:0}#title-block-header.quarto-title-block.default .quarto-title-meta-contents p,#title-block-header.quarto-title-block.default .quarto-title-authors p,#title-block-header.quarto-title-block.default .quarto-title-affiliations p{margin-bottom:.1em}#title-block-header.quarto-title-block.default .quarto-title-meta-heading{text-transform:uppercase;margin-top:1em;font-size:.8em;opacity:.8;font-weight:400}#title-block-header.quarto-title-block.default .quarto-title-meta-contents{font-size:.9em}#title-block-header.quarto-title-block.default .quarto-title-meta-contents a{color:#333}#title-block-header.quarto-title-block.default .quarto-title-meta-contents p.affiliation:last-of-type{margin-bottom:.7em}#title-block-header.quarto-title-block.default p.affiliation{margin-bottom:.1em}#title-block-header.quarto-title-block.default .description,#title-block-header.quarto-title-block.default .abstract{margin-top:0}#title-block-header.quarto-title-block.default .description>p,#title-block-header.quarto-title-block.default .abstract>p{font-size:.9em}#title-block-header.quarto-title-block.default .description>p:last-of-type,#title-block-header.quarto-title-block.default .abstract>p:last-of-type{margin-bottom:0}#title-block-header.quarto-title-block.default .description .abstract-title,#title-block-header.quarto-title-block.default .abstract .abstract-title{margin-top:1em;text-transform:uppercase;font-size:.8em;opacity:.8;font-weight:400}#title-block-header.quarto-title-block.default .quarto-title-meta-author{display:grid;grid-template-columns:1fr 1fr}.quarto-title-tools-only{display:flex;justify-content:right}/*# sourceMappingURL=a9ddc644a24cafe999652d0a1861bbee.css.map */
+*/.ansi-black-fg{color:#3e424d}.ansi-black-bg{background-color:#3e424d}.ansi-black-intense-fg{color:#282c36}.ansi-black-intense-bg{background-color:#282c36}.ansi-red-fg{color:#e75c58}.ansi-red-bg{background-color:#e75c58}.ansi-red-intense-fg{color:#b22b31}.ansi-red-intense-bg{background-color:#b22b31}.ansi-green-fg{color:#00a250}.ansi-green-bg{background-color:#00a250}.ansi-green-intense-fg{color:#007427}.ansi-green-intense-bg{background-color:#007427}.ansi-yellow-fg{color:#ddb62b}.ansi-yellow-bg{background-color:#ddb62b}.ansi-yellow-intense-fg{color:#b27d12}.ansi-yellow-intense-bg{background-color:#b27d12}.ansi-blue-fg{color:#208ffb}.ansi-blue-bg{background-color:#208ffb}.ansi-blue-intense-fg{color:#0065ca}.ansi-blue-intense-bg{background-color:#0065ca}.ansi-magenta-fg{color:#d160c4}.ansi-magenta-bg{background-color:#d160c4}.ansi-magenta-intense-fg{color:#a03196}.ansi-magenta-intense-bg{background-color:#a03196}.ansi-cyan-fg{color:#60c6c8}.ansi-cyan-bg{background-color:#60c6c8}.ansi-cyan-intense-fg{color:#258f8f}.ansi-cyan-intense-bg{background-color:#258f8f}.ansi-white-fg{color:#c5c1b4}.ansi-white-bg{background-color:#c5c1b4}.ansi-white-intense-fg{color:#a1a6b2}.ansi-white-intense-bg{background-color:#a1a6b2}.ansi-default-inverse-fg{color:#fff}.ansi-default-inverse-bg{background-color:#000}.ansi-bold{font-weight:bold}.ansi-underline{text-decoration:underline}:root{--quarto-body-bg: #fff;--quarto-body-color: #333;--quarto-text-muted: #6c757d;--quarto-border-color: #dee2e6;--quarto-border-width: 1px;--quarto-border-radius: 0.25rem}table.gt_table{color:var(--quarto-body-color);font-size:1em;width:100%;background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_column_spanner_outer{color:var(--quarto-body-color);background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_col_heading{color:var(--quarto-body-color);font-weight:bold;background-color:rgba(0,0,0,0)}table.gt_table thead.gt_col_headings{border-bottom:1px solid currentColor;border-top-width:inherit;border-top-color:var(--quarto-border-color)}table.gt_table thead.gt_col_headings:not(:first-child){border-top-width:1px;border-top-color:var(--quarto-border-color)}table.gt_table td.gt_row{border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-width:0px}table.gt_table tbody.gt_table_body{border-top-width:1px;border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-color:currentColor}div.columns{display:initial;gap:initial}div.column{display:inline-block;overflow-x:initial;vertical-align:top;width:50%}.code-annotation-tip-content{word-wrap:break-word}.code-annotation-container-hidden{display:none !important}dl.code-annotation-container-grid{display:grid;grid-template-columns:min-content auto}dl.code-annotation-container-grid dt{grid-column:1}dl.code-annotation-container-grid dd{grid-column:2}pre.sourceCode.code-annotation-code{padding-right:0}code.sourceCode .code-annotation-anchor{z-index:100;position:absolute;right:.5em;left:inherit;background-color:rgba(0,0,0,0)}:root{--mermaid-bg-color: #fff;--mermaid-edge-color: #adb5bd;--mermaid-node-fg-color: #333;--mermaid-fg-color: #333;--mermaid-fg-color--lighter: #4d4d4d;--mermaid-fg-color--lightest: #666666;--mermaid-font-family: Ubuntu, -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol;--mermaid-label-bg-color: #fff;--mermaid-label-fg-color: #e95420;--mermaid-node-bg-color: rgba(233, 84, 32, 0.1);--mermaid-node-fg-color: #333}@media print{:root{font-size:11pt}#quarto-sidebar,#TOC,.nav-page{display:none}.page-columns .content{grid-column-start:page-start}.fixed-top{position:relative}.panel-caption,.figure-caption,figcaption{color:#666}}.code-copy-button{position:absolute;top:0;right:0;border:0;margin-top:5px;margin-right:5px;background-color:rgba(0,0,0,0);z-index:3}.code-copy-button:focus{outline:none}.code-copy-button-tooltip{font-size:.75em}.code-copy-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}.code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}.code-copy-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}.code-copy-button-checked:hover>.bi::before{background-image:url('data:image/svg+xml,')}main ol ol,main ul ul,main ol ul,main ul ol{margin-bottom:1em}ul>li:not(:has(>p))>ul,ol>li:not(:has(>p))>ul,ul>li:not(:has(>p))>ol,ol>li:not(:has(>p))>ol{margin-bottom:0}ul>li:not(:has(>p))>ul>li:has(>p),ol>li:not(:has(>p))>ul>li:has(>p),ul>li:not(:has(>p))>ol>li:has(>p),ol>li:not(:has(>p))>ol>li:has(>p){margin-top:1rem}body{margin:0}main.page-columns>header>h1.title,main.page-columns>header>.title.h1{margin-bottom:0}@media(min-width: 992px){body .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.fullcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] 35px [page-end-inset page-end] 5fr [screen-end-inset] 1.5em}body.slimcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.listing:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 3em [body-end] 50px [body-end-outset] minmax(0px, 250px) [page-end-inset] minmax(50px, 100px) [page-end] 1fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 175px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 175px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] minmax(25px, 50px) [page-start-inset] minmax(50px, 150px) [body-start-outset] minmax(25px, 50px) [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] minmax(25px, 50px) [body-end-outset] minmax(50px, 150px) [page-end-inset] minmax(25px, 50px) [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(50px, 100px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 50px [page-start-inset] minmax(50px, 150px) [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(450px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 50px [page-start-inset] minmax(50px, 150px) [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(450px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(50px, 150px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] minmax(25px, 50px) [page-start-inset] minmax(50px, 150px) [body-start-outset] minmax(25px, 50px) [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] minmax(25px, 50px) [body-end-outset] minmax(50px, 150px) [page-end-inset] minmax(25px, 50px) [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}}@media(max-width: 991.98px){body .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.fullcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.slimcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.listing:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 1250px - 3em )) [body-content-end body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 145px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 145px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1.5em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(75px, 150px) [page-end-inset] 25px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 4fr [screen-end-inset] 1.5em [screen-end]}body.docked.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 4fr [screen-end-inset] 1.5em [screen-end]}body.floating.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(75px, 150px) [page-end-inset] 25px [page-end] 4fr [screen-end-inset] 1.5em [screen-end]}}@media(max-width: 767.98px){body .page-columns,body.fullcontent:not(.floating):not(.docked) .page-columns,body.slimcontent:not(.floating):not(.docked) .page-columns,body.docked .page-columns,body.docked.slimcontent .page-columns,body.docked.fullcontent .page-columns,body.floating .page-columns,body.floating.slimcontent .page-columns,body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}nav[role=doc-toc]{display:none}}body,.page-row-navigation{grid-template-rows:[page-top] max-content [contents-top] max-content [contents-bottom] max-content [page-bottom]}.page-rows-contents{grid-template-rows:[content-top] minmax(max-content, 1fr) [content-bottom] minmax(60px, max-content) [page-bottom]}.page-full{grid-column:screen-start/screen-end !important}.page-columns>*{grid-column:body-content-start/body-content-end}.page-columns.column-page>*{grid-column:page-start/page-end}.page-columns.column-page-left>*{grid-column:page-start/body-content-end}.page-columns.column-page-right>*{grid-column:body-content-start/page-end}.page-rows{grid-auto-rows:auto}.header{grid-column:screen-start/screen-end;grid-row:page-top/contents-top}#quarto-content{padding:0;grid-column:screen-start/screen-end;grid-row:contents-top/contents-bottom}body.floating .sidebar.sidebar-navigation{grid-column:page-start/body-start;grid-row:content-top/page-bottom}body.docked .sidebar.sidebar-navigation{grid-column:screen-start/body-start;grid-row:content-top/page-bottom}.sidebar.toc-left{grid-column:page-start/body-start;grid-row:content-top/page-bottom}.sidebar.margin-sidebar{grid-column:body-end/page-end;grid-row:content-top/page-bottom}.page-columns .content{grid-column:body-content-start/body-content-end;grid-row:content-top/content-bottom;align-content:flex-start}.page-columns .page-navigation{grid-column:body-content-start/body-content-end;grid-row:content-bottom/page-bottom}.page-columns .footer{grid-column:screen-start/screen-end;grid-row:contents-bottom/page-bottom}.page-columns .column-body{grid-column:body-content-start/body-content-end}.page-columns .column-body-fullbleed{grid-column:body-start/body-end}.page-columns .column-body-outset{grid-column:body-start-outset/body-end-outset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset table{background:#fff}.page-columns .column-body-outset-left{grid-column:body-start-outset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset-left table{background:#fff}.page-columns .column-body-outset-right{grid-column:body-content-start/body-end-outset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset-right table{background:#fff}.page-columns .column-page{grid-column:page-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page table{background:#fff}.page-columns .column-page-inset{grid-column:page-start-inset/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset table{background:#fff}.page-columns .column-page-inset-left{grid-column:page-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset-left table{background:#fff}.page-columns .column-page-inset-right{grid-column:body-content-start/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset-right figcaption table{background:#fff}.page-columns .column-page-left{grid-column:page-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-left table{background:#fff}.page-columns .column-page-right{grid-column:body-content-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-right figcaption table{background:#fff}#quarto-content.page-columns #quarto-margin-sidebar,#quarto-content.page-columns #quarto-sidebar{z-index:1}@media(max-width: 991.98px){#quarto-content.page-columns #quarto-margin-sidebar.collapse,#quarto-content.page-columns #quarto-sidebar.collapse,#quarto-content.page-columns #quarto-margin-sidebar.collapsing,#quarto-content.page-columns #quarto-sidebar.collapsing{z-index:1055}}#quarto-content.page-columns main.column-page,#quarto-content.page-columns main.column-page-right,#quarto-content.page-columns main.column-page-left{z-index:0}.page-columns .column-screen-inset{grid-column:screen-start-inset/screen-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset table{background:#fff}.page-columns .column-screen-inset-left{grid-column:screen-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-left table{background:#fff}.page-columns .column-screen-inset-right{grid-column:body-content-start/screen-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-right table{background:#fff}.page-columns .column-screen{grid-column:screen-start/screen-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen table{background:#fff}.page-columns .column-screen-left{grid-column:screen-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-left table{background:#fff}.page-columns .column-screen-right{grid-column:body-content-start/screen-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-right table{background:#fff}.page-columns .column-screen-inset-shaded{grid-column:screen-start/screen-end;padding:1em;background:#e9ecef;z-index:998;transform:translate3d(0, 0, 0);margin-bottom:1em}.zindex-content{z-index:998;transform:translate3d(0, 0, 0)}.zindex-modal{z-index:1055;transform:translate3d(0, 0, 0)}.zindex-over-content{z-index:999;transform:translate3d(0, 0, 0)}img.img-fluid.column-screen,img.img-fluid.column-screen-inset-shaded,img.img-fluid.column-screen-inset,img.img-fluid.column-screen-inset-left,img.img-fluid.column-screen-inset-right,img.img-fluid.column-screen-left,img.img-fluid.column-screen-right{width:100%}@media(min-width: 992px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-end/page-end !important;z-index:998}.column-sidebar{grid-column:page-start/body-start !important;z-index:998}.column-leftmargin{grid-column:screen-start-inset/body-start !important;z-index:998}.no-row-height{height:1em;overflow:visible}}@media(max-width: 991.98px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-end/page-end !important;z-index:998}.no-row-height{height:1em;overflow:visible}.page-columns.page-full{overflow:visible}.page-columns.toc-left .margin-caption,.page-columns.toc-left div.aside,.page-columns.toc-left aside,.page-columns.toc-left .column-margin{grid-column:body-content-start/body-content-end !important;z-index:998;transform:translate3d(0, 0, 0)}.page-columns.toc-left .no-row-height{height:initial;overflow:initial}}@media(max-width: 767.98px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-content-start/body-content-end !important;z-index:998;transform:translate3d(0, 0, 0)}.no-row-height{height:initial;overflow:initial}#quarto-margin-sidebar{display:none}#quarto-sidebar-toc-left{display:none}.hidden-sm{display:none}}.panel-grid{display:grid;grid-template-rows:repeat(1, 1fr);grid-template-columns:repeat(24, 1fr);gap:1em}.panel-grid .g-col-1{grid-column:auto/span 1}.panel-grid .g-col-2{grid-column:auto/span 2}.panel-grid .g-col-3{grid-column:auto/span 3}.panel-grid .g-col-4{grid-column:auto/span 4}.panel-grid .g-col-5{grid-column:auto/span 5}.panel-grid .g-col-6{grid-column:auto/span 6}.panel-grid .g-col-7{grid-column:auto/span 7}.panel-grid .g-col-8{grid-column:auto/span 8}.panel-grid .g-col-9{grid-column:auto/span 9}.panel-grid .g-col-10{grid-column:auto/span 10}.panel-grid .g-col-11{grid-column:auto/span 11}.panel-grid .g-col-12{grid-column:auto/span 12}.panel-grid .g-col-13{grid-column:auto/span 13}.panel-grid .g-col-14{grid-column:auto/span 14}.panel-grid .g-col-15{grid-column:auto/span 15}.panel-grid .g-col-16{grid-column:auto/span 16}.panel-grid .g-col-17{grid-column:auto/span 17}.panel-grid .g-col-18{grid-column:auto/span 18}.panel-grid .g-col-19{grid-column:auto/span 19}.panel-grid .g-col-20{grid-column:auto/span 20}.panel-grid .g-col-21{grid-column:auto/span 21}.panel-grid .g-col-22{grid-column:auto/span 22}.panel-grid .g-col-23{grid-column:auto/span 23}.panel-grid .g-col-24{grid-column:auto/span 24}.panel-grid .g-start-1{grid-column-start:1}.panel-grid .g-start-2{grid-column-start:2}.panel-grid .g-start-3{grid-column-start:3}.panel-grid .g-start-4{grid-column-start:4}.panel-grid .g-start-5{grid-column-start:5}.panel-grid .g-start-6{grid-column-start:6}.panel-grid .g-start-7{grid-column-start:7}.panel-grid .g-start-8{grid-column-start:8}.panel-grid .g-start-9{grid-column-start:9}.panel-grid .g-start-10{grid-column-start:10}.panel-grid .g-start-11{grid-column-start:11}.panel-grid .g-start-12{grid-column-start:12}.panel-grid .g-start-13{grid-column-start:13}.panel-grid .g-start-14{grid-column-start:14}.panel-grid .g-start-15{grid-column-start:15}.panel-grid .g-start-16{grid-column-start:16}.panel-grid .g-start-17{grid-column-start:17}.panel-grid .g-start-18{grid-column-start:18}.panel-grid .g-start-19{grid-column-start:19}.panel-grid .g-start-20{grid-column-start:20}.panel-grid .g-start-21{grid-column-start:21}.panel-grid .g-start-22{grid-column-start:22}.panel-grid .g-start-23{grid-column-start:23}@media(min-width: 576px){.panel-grid .g-col-sm-1{grid-column:auto/span 1}.panel-grid .g-col-sm-2{grid-column:auto/span 2}.panel-grid .g-col-sm-3{grid-column:auto/span 3}.panel-grid .g-col-sm-4{grid-column:auto/span 4}.panel-grid .g-col-sm-5{grid-column:auto/span 5}.panel-grid .g-col-sm-6{grid-column:auto/span 6}.panel-grid .g-col-sm-7{grid-column:auto/span 7}.panel-grid .g-col-sm-8{grid-column:auto/span 8}.panel-grid .g-col-sm-9{grid-column:auto/span 9}.panel-grid .g-col-sm-10{grid-column:auto/span 10}.panel-grid .g-col-sm-11{grid-column:auto/span 11}.panel-grid .g-col-sm-12{grid-column:auto/span 12}.panel-grid .g-col-sm-13{grid-column:auto/span 13}.panel-grid .g-col-sm-14{grid-column:auto/span 14}.panel-grid .g-col-sm-15{grid-column:auto/span 15}.panel-grid .g-col-sm-16{grid-column:auto/span 16}.panel-grid .g-col-sm-17{grid-column:auto/span 17}.panel-grid .g-col-sm-18{grid-column:auto/span 18}.panel-grid .g-col-sm-19{grid-column:auto/span 19}.panel-grid .g-col-sm-20{grid-column:auto/span 20}.panel-grid .g-col-sm-21{grid-column:auto/span 21}.panel-grid .g-col-sm-22{grid-column:auto/span 22}.panel-grid .g-col-sm-23{grid-column:auto/span 23}.panel-grid .g-col-sm-24{grid-column:auto/span 24}.panel-grid .g-start-sm-1{grid-column-start:1}.panel-grid .g-start-sm-2{grid-column-start:2}.panel-grid .g-start-sm-3{grid-column-start:3}.panel-grid .g-start-sm-4{grid-column-start:4}.panel-grid .g-start-sm-5{grid-column-start:5}.panel-grid .g-start-sm-6{grid-column-start:6}.panel-grid .g-start-sm-7{grid-column-start:7}.panel-grid .g-start-sm-8{grid-column-start:8}.panel-grid .g-start-sm-9{grid-column-start:9}.panel-grid .g-start-sm-10{grid-column-start:10}.panel-grid .g-start-sm-11{grid-column-start:11}.panel-grid .g-start-sm-12{grid-column-start:12}.panel-grid .g-start-sm-13{grid-column-start:13}.panel-grid .g-start-sm-14{grid-column-start:14}.panel-grid .g-start-sm-15{grid-column-start:15}.panel-grid .g-start-sm-16{grid-column-start:16}.panel-grid .g-start-sm-17{grid-column-start:17}.panel-grid .g-start-sm-18{grid-column-start:18}.panel-grid .g-start-sm-19{grid-column-start:19}.panel-grid .g-start-sm-20{grid-column-start:20}.panel-grid .g-start-sm-21{grid-column-start:21}.panel-grid .g-start-sm-22{grid-column-start:22}.panel-grid .g-start-sm-23{grid-column-start:23}}@media(min-width: 768px){.panel-grid .g-col-md-1{grid-column:auto/span 1}.panel-grid .g-col-md-2{grid-column:auto/span 2}.panel-grid .g-col-md-3{grid-column:auto/span 3}.panel-grid .g-col-md-4{grid-column:auto/span 4}.panel-grid .g-col-md-5{grid-column:auto/span 5}.panel-grid .g-col-md-6{grid-column:auto/span 6}.panel-grid .g-col-md-7{grid-column:auto/span 7}.panel-grid .g-col-md-8{grid-column:auto/span 8}.panel-grid .g-col-md-9{grid-column:auto/span 9}.panel-grid .g-col-md-10{grid-column:auto/span 10}.panel-grid .g-col-md-11{grid-column:auto/span 11}.panel-grid .g-col-md-12{grid-column:auto/span 12}.panel-grid .g-col-md-13{grid-column:auto/span 13}.panel-grid .g-col-md-14{grid-column:auto/span 14}.panel-grid .g-col-md-15{grid-column:auto/span 15}.panel-grid .g-col-md-16{grid-column:auto/span 16}.panel-grid .g-col-md-17{grid-column:auto/span 17}.panel-grid .g-col-md-18{grid-column:auto/span 18}.panel-grid .g-col-md-19{grid-column:auto/span 19}.panel-grid .g-col-md-20{grid-column:auto/span 20}.panel-grid .g-col-md-21{grid-column:auto/span 21}.panel-grid .g-col-md-22{grid-column:auto/span 22}.panel-grid .g-col-md-23{grid-column:auto/span 23}.panel-grid .g-col-md-24{grid-column:auto/span 24}.panel-grid .g-start-md-1{grid-column-start:1}.panel-grid .g-start-md-2{grid-column-start:2}.panel-grid .g-start-md-3{grid-column-start:3}.panel-grid .g-start-md-4{grid-column-start:4}.panel-grid .g-start-md-5{grid-column-start:5}.panel-grid .g-start-md-6{grid-column-start:6}.panel-grid .g-start-md-7{grid-column-start:7}.panel-grid .g-start-md-8{grid-column-start:8}.panel-grid .g-start-md-9{grid-column-start:9}.panel-grid .g-start-md-10{grid-column-start:10}.panel-grid .g-start-md-11{grid-column-start:11}.panel-grid .g-start-md-12{grid-column-start:12}.panel-grid .g-start-md-13{grid-column-start:13}.panel-grid .g-start-md-14{grid-column-start:14}.panel-grid .g-start-md-15{grid-column-start:15}.panel-grid .g-start-md-16{grid-column-start:16}.panel-grid .g-start-md-17{grid-column-start:17}.panel-grid .g-start-md-18{grid-column-start:18}.panel-grid .g-start-md-19{grid-column-start:19}.panel-grid .g-start-md-20{grid-column-start:20}.panel-grid .g-start-md-21{grid-column-start:21}.panel-grid .g-start-md-22{grid-column-start:22}.panel-grid .g-start-md-23{grid-column-start:23}}@media(min-width: 992px){.panel-grid .g-col-lg-1{grid-column:auto/span 1}.panel-grid .g-col-lg-2{grid-column:auto/span 2}.panel-grid .g-col-lg-3{grid-column:auto/span 3}.panel-grid .g-col-lg-4{grid-column:auto/span 4}.panel-grid .g-col-lg-5{grid-column:auto/span 5}.panel-grid .g-col-lg-6{grid-column:auto/span 6}.panel-grid .g-col-lg-7{grid-column:auto/span 7}.panel-grid .g-col-lg-8{grid-column:auto/span 8}.panel-grid .g-col-lg-9{grid-column:auto/span 9}.panel-grid .g-col-lg-10{grid-column:auto/span 10}.panel-grid .g-col-lg-11{grid-column:auto/span 11}.panel-grid .g-col-lg-12{grid-column:auto/span 12}.panel-grid .g-col-lg-13{grid-column:auto/span 13}.panel-grid .g-col-lg-14{grid-column:auto/span 14}.panel-grid .g-col-lg-15{grid-column:auto/span 15}.panel-grid .g-col-lg-16{grid-column:auto/span 16}.panel-grid .g-col-lg-17{grid-column:auto/span 17}.panel-grid .g-col-lg-18{grid-column:auto/span 18}.panel-grid .g-col-lg-19{grid-column:auto/span 19}.panel-grid .g-col-lg-20{grid-column:auto/span 20}.panel-grid .g-col-lg-21{grid-column:auto/span 21}.panel-grid .g-col-lg-22{grid-column:auto/span 22}.panel-grid .g-col-lg-23{grid-column:auto/span 23}.panel-grid .g-col-lg-24{grid-column:auto/span 24}.panel-grid .g-start-lg-1{grid-column-start:1}.panel-grid .g-start-lg-2{grid-column-start:2}.panel-grid .g-start-lg-3{grid-column-start:3}.panel-grid .g-start-lg-4{grid-column-start:4}.panel-grid .g-start-lg-5{grid-column-start:5}.panel-grid .g-start-lg-6{grid-column-start:6}.panel-grid .g-start-lg-7{grid-column-start:7}.panel-grid .g-start-lg-8{grid-column-start:8}.panel-grid .g-start-lg-9{grid-column-start:9}.panel-grid .g-start-lg-10{grid-column-start:10}.panel-grid .g-start-lg-11{grid-column-start:11}.panel-grid .g-start-lg-12{grid-column-start:12}.panel-grid .g-start-lg-13{grid-column-start:13}.panel-grid .g-start-lg-14{grid-column-start:14}.panel-grid .g-start-lg-15{grid-column-start:15}.panel-grid .g-start-lg-16{grid-column-start:16}.panel-grid .g-start-lg-17{grid-column-start:17}.panel-grid .g-start-lg-18{grid-column-start:18}.panel-grid .g-start-lg-19{grid-column-start:19}.panel-grid .g-start-lg-20{grid-column-start:20}.panel-grid .g-start-lg-21{grid-column-start:21}.panel-grid .g-start-lg-22{grid-column-start:22}.panel-grid .g-start-lg-23{grid-column-start:23}}@media(min-width: 1200px){.panel-grid .g-col-xl-1{grid-column:auto/span 1}.panel-grid .g-col-xl-2{grid-column:auto/span 2}.panel-grid .g-col-xl-3{grid-column:auto/span 3}.panel-grid .g-col-xl-4{grid-column:auto/span 4}.panel-grid .g-col-xl-5{grid-column:auto/span 5}.panel-grid .g-col-xl-6{grid-column:auto/span 6}.panel-grid .g-col-xl-7{grid-column:auto/span 7}.panel-grid .g-col-xl-8{grid-column:auto/span 8}.panel-grid .g-col-xl-9{grid-column:auto/span 9}.panel-grid .g-col-xl-10{grid-column:auto/span 10}.panel-grid .g-col-xl-11{grid-column:auto/span 11}.panel-grid .g-col-xl-12{grid-column:auto/span 12}.panel-grid .g-col-xl-13{grid-column:auto/span 13}.panel-grid .g-col-xl-14{grid-column:auto/span 14}.panel-grid .g-col-xl-15{grid-column:auto/span 15}.panel-grid .g-col-xl-16{grid-column:auto/span 16}.panel-grid .g-col-xl-17{grid-column:auto/span 17}.panel-grid .g-col-xl-18{grid-column:auto/span 18}.panel-grid .g-col-xl-19{grid-column:auto/span 19}.panel-grid .g-col-xl-20{grid-column:auto/span 20}.panel-grid .g-col-xl-21{grid-column:auto/span 21}.panel-grid .g-col-xl-22{grid-column:auto/span 22}.panel-grid .g-col-xl-23{grid-column:auto/span 23}.panel-grid .g-col-xl-24{grid-column:auto/span 24}.panel-grid .g-start-xl-1{grid-column-start:1}.panel-grid .g-start-xl-2{grid-column-start:2}.panel-grid .g-start-xl-3{grid-column-start:3}.panel-grid .g-start-xl-4{grid-column-start:4}.panel-grid .g-start-xl-5{grid-column-start:5}.panel-grid .g-start-xl-6{grid-column-start:6}.panel-grid .g-start-xl-7{grid-column-start:7}.panel-grid .g-start-xl-8{grid-column-start:8}.panel-grid .g-start-xl-9{grid-column-start:9}.panel-grid .g-start-xl-10{grid-column-start:10}.panel-grid .g-start-xl-11{grid-column-start:11}.panel-grid .g-start-xl-12{grid-column-start:12}.panel-grid .g-start-xl-13{grid-column-start:13}.panel-grid .g-start-xl-14{grid-column-start:14}.panel-grid .g-start-xl-15{grid-column-start:15}.panel-grid .g-start-xl-16{grid-column-start:16}.panel-grid .g-start-xl-17{grid-column-start:17}.panel-grid .g-start-xl-18{grid-column-start:18}.panel-grid .g-start-xl-19{grid-column-start:19}.panel-grid .g-start-xl-20{grid-column-start:20}.panel-grid .g-start-xl-21{grid-column-start:21}.panel-grid .g-start-xl-22{grid-column-start:22}.panel-grid .g-start-xl-23{grid-column-start:23}}@media(min-width: 1400px){.panel-grid .g-col-xxl-1{grid-column:auto/span 1}.panel-grid .g-col-xxl-2{grid-column:auto/span 2}.panel-grid .g-col-xxl-3{grid-column:auto/span 3}.panel-grid .g-col-xxl-4{grid-column:auto/span 4}.panel-grid .g-col-xxl-5{grid-column:auto/span 5}.panel-grid .g-col-xxl-6{grid-column:auto/span 6}.panel-grid .g-col-xxl-7{grid-column:auto/span 7}.panel-grid .g-col-xxl-8{grid-column:auto/span 8}.panel-grid .g-col-xxl-9{grid-column:auto/span 9}.panel-grid .g-col-xxl-10{grid-column:auto/span 10}.panel-grid .g-col-xxl-11{grid-column:auto/span 11}.panel-grid .g-col-xxl-12{grid-column:auto/span 12}.panel-grid .g-col-xxl-13{grid-column:auto/span 13}.panel-grid .g-col-xxl-14{grid-column:auto/span 14}.panel-grid .g-col-xxl-15{grid-column:auto/span 15}.panel-grid .g-col-xxl-16{grid-column:auto/span 16}.panel-grid .g-col-xxl-17{grid-column:auto/span 17}.panel-grid .g-col-xxl-18{grid-column:auto/span 18}.panel-grid .g-col-xxl-19{grid-column:auto/span 19}.panel-grid .g-col-xxl-20{grid-column:auto/span 20}.panel-grid .g-col-xxl-21{grid-column:auto/span 21}.panel-grid .g-col-xxl-22{grid-column:auto/span 22}.panel-grid .g-col-xxl-23{grid-column:auto/span 23}.panel-grid .g-col-xxl-24{grid-column:auto/span 24}.panel-grid .g-start-xxl-1{grid-column-start:1}.panel-grid .g-start-xxl-2{grid-column-start:2}.panel-grid .g-start-xxl-3{grid-column-start:3}.panel-grid .g-start-xxl-4{grid-column-start:4}.panel-grid .g-start-xxl-5{grid-column-start:5}.panel-grid .g-start-xxl-6{grid-column-start:6}.panel-grid .g-start-xxl-7{grid-column-start:7}.panel-grid .g-start-xxl-8{grid-column-start:8}.panel-grid .g-start-xxl-9{grid-column-start:9}.panel-grid .g-start-xxl-10{grid-column-start:10}.panel-grid .g-start-xxl-11{grid-column-start:11}.panel-grid .g-start-xxl-12{grid-column-start:12}.panel-grid .g-start-xxl-13{grid-column-start:13}.panel-grid .g-start-xxl-14{grid-column-start:14}.panel-grid .g-start-xxl-15{grid-column-start:15}.panel-grid .g-start-xxl-16{grid-column-start:16}.panel-grid .g-start-xxl-17{grid-column-start:17}.panel-grid .g-start-xxl-18{grid-column-start:18}.panel-grid .g-start-xxl-19{grid-column-start:19}.panel-grid .g-start-xxl-20{grid-column-start:20}.panel-grid .g-start-xxl-21{grid-column-start:21}.panel-grid .g-start-xxl-22{grid-column-start:22}.panel-grid .g-start-xxl-23{grid-column-start:23}}main{margin-top:1em;margin-bottom:1em}h1,.h1,h2,.h2{opacity:.9;margin-top:2rem;margin-bottom:1rem;font-weight:600}h1.title,.title.h1{margin-top:0}h2,.h2{border-bottom:1px solid #dee2e6;padding-bottom:.5rem}h3,.h3{font-weight:600}h3,.h3,h4,.h4{opacity:.9;margin-top:1.5rem}h5,.h5,h6,.h6{opacity:.9}.header-section-number{color:#737373}.nav-link.active .header-section-number{color:inherit}mark,.mark{padding:0em}.panel-caption,caption,.figure-caption{font-size:.9rem}.panel-caption,.figure-caption,figcaption{color:#737373}.table-caption,caption{color:#333}.quarto-layout-cell[data-ref-parent] caption{color:#737373}.column-margin figcaption,.margin-caption,div.aside,aside,.column-margin{color:#737373;font-size:.825rem}.panel-caption.margin-caption{text-align:inherit}.column-margin.column-container p{margin-bottom:0}.column-margin.column-container>*:not(.collapse){padding-top:.5em;padding-bottom:.5em;display:block}.column-margin.column-container>*.collapse:not(.show){display:none}@media(min-width: 768px){.column-margin.column-container .callout-margin-content:first-child{margin-top:4.5em}.column-margin.column-container .callout-margin-content-simple:first-child{margin-top:3.5em}}.margin-caption>*{padding-top:.5em;padding-bottom:.5em}@media(max-width: 767.98px){.quarto-layout-row{flex-direction:column}}.nav-tabs .nav-item{margin-top:1px;cursor:pointer}.tab-content{margin-top:0px;border-left:#dee2e6 1px solid;border-right:#dee2e6 1px solid;border-bottom:#dee2e6 1px solid;margin-left:0;padding:1em;margin-bottom:1em}@media(max-width: 767.98px){.layout-sidebar{margin-left:0;margin-right:0}}.panel-sidebar,.panel-sidebar .form-control,.panel-input,.panel-input .form-control,.selectize-dropdown{font-size:.9rem}.panel-sidebar .form-control,.panel-input .form-control{padding-top:.1rem}.tab-pane div.sourceCode{margin-top:0px}.tab-pane>p{padding-top:1em}.tab-content>.tab-pane:not(.active){display:none !important}div.sourceCode{background-color:rgba(233,236,239,.65);border:1px solid rgba(233,236,239,.65);border-radius:.25rem}pre.sourceCode{background-color:rgba(0,0,0,0)}pre.sourceCode{border:none;font-size:.875em;overflow:visible !important;padding:.4em}.callout pre.sourceCode{padding-left:0}div.sourceCode{overflow-y:hidden}.callout div.sourceCode{margin-left:initial}.blockquote{font-size:inherit;padding-left:1rem;padding-right:1.5rem;color:#737373}.blockquote h1:first-child,.blockquote .h1:first-child,.blockquote h2:first-child,.blockquote .h2:first-child,.blockquote h3:first-child,.blockquote .h3:first-child,.blockquote h4:first-child,.blockquote .h4:first-child,.blockquote h5:first-child,.blockquote .h5:first-child{margin-top:0}pre{background-color:initial;padding:initial;border:initial}p code:not(.sourceCode),li code:not(.sourceCode),td code:not(.sourceCode){background-color:#f7f7f7;padding:.2em}nav p code:not(.sourceCode),nav li code:not(.sourceCode),nav td code:not(.sourceCode){background-color:rgba(0,0,0,0);padding:0}td code:not(.sourceCode){white-space:pre-wrap}#quarto-embedded-source-code-modal>.modal-dialog{max-width:1000px;padding-left:1.75rem;padding-right:1.75rem}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-body{padding:0}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-body div.sourceCode{margin:0;padding:.2rem .2rem;border-radius:0px;border:none}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-header{padding:.7rem}.code-tools-button{font-size:1rem;padding:.15rem .15rem;margin-left:5px;color:#6c757d;background-color:rgba(0,0,0,0);transition:initial;cursor:pointer}.code-tools-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}.code-tools-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}#quarto-embedded-source-code-modal .code-copy-button>.bi::before{background-image:url('data:image/svg+xml,')}#quarto-embedded-source-code-modal .code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}.sidebar{will-change:top;transition:top 200ms linear;position:sticky;overflow-y:auto;padding-top:1.2em;max-height:100vh}.sidebar.toc-left,.sidebar.margin-sidebar{top:0px;padding-top:1em}.sidebar.toc-left>*,.sidebar.margin-sidebar>*{padding-top:.5em}.sidebar.quarto-banner-title-block-sidebar>*{padding-top:1.65em}figure .quarto-notebook-link{margin-top:.5em}.quarto-notebook-link{font-size:.75em;color:#6c757d;margin-bottom:1em;text-decoration:none;display:block}.quarto-notebook-link:hover{text-decoration:underline;color:#e95420}.quarto-notebook-link::before{display:inline-block;height:.75rem;width:.75rem;margin-bottom:0em;margin-right:.25em;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:.75rem .75rem}.quarto-alternate-notebooks i.bi,.quarto-alternate-formats i.bi{margin-right:.4em}.quarto-notebook .cell-container{display:flex}.quarto-notebook .cell-container .cell{flex-grow:4}.quarto-notebook .cell-container .cell-decorator{padding-top:1.5em;padding-right:1em;text-align:right}.quarto-notebook h2,.quarto-notebook .h2{border-bottom:none}.sidebar .quarto-alternate-formats a,.sidebar .quarto-alternate-notebooks a{text-decoration:none}.sidebar .quarto-alternate-formats a:hover,.sidebar .quarto-alternate-notebooks a:hover{color:#e95420}.sidebar .quarto-alternate-notebooks h2,.sidebar .quarto-alternate-notebooks .h2,.sidebar .quarto-alternate-formats h2,.sidebar .quarto-alternate-formats .h2,.sidebar nav[role=doc-toc]>h2,.sidebar nav[role=doc-toc]>.h2{font-size:.875rem;font-weight:400;margin-bottom:.5rem;margin-top:.3rem;font-family:inherit;border-bottom:0;padding-bottom:0;padding-top:0px}.sidebar .quarto-alternate-notebooks h2,.sidebar .quarto-alternate-notebooks .h2,.sidebar .quarto-alternate-formats h2,.sidebar .quarto-alternate-formats .h2{margin-top:1rem}.sidebar nav[role=doc-toc]>ul a{border-left:1px solid #e9ecef;padding-left:.6rem}.sidebar .quarto-alternate-notebooks h2>ul a,.sidebar .quarto-alternate-notebooks .h2>ul a,.sidebar .quarto-alternate-formats h2>ul a,.sidebar .quarto-alternate-formats .h2>ul a{border-left:none;padding-left:.6rem}.sidebar .quarto-alternate-notebooks ul a:empty,.sidebar .quarto-alternate-formats ul a:empty,.sidebar nav[role=doc-toc]>ul a:empty{display:none}.sidebar .quarto-alternate-notebooks ul,.sidebar .quarto-alternate-formats ul,.sidebar nav[role=doc-toc] ul{padding-left:0;list-style:none;font-size:.875rem;font-weight:300}.sidebar .quarto-alternate-notebooks ul li a,.sidebar .quarto-alternate-formats ul li a,.sidebar nav[role=doc-toc]>ul li a{line-height:1.1rem;padding-bottom:.2rem;padding-top:.2rem;color:inherit}.sidebar nav[role=doc-toc] ul>li>ul>li>a{padding-left:1.2em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>a{padding-left:2.4em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>a{padding-left:3.6em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>ul>li>a{padding-left:4.8em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>ul>li>ul>li>a{padding-left:6em}.sidebar nav[role=doc-toc] ul>li>a.active,.sidebar nav[role=doc-toc] ul>li>ul>li>a.active{border-left:1px solid #e95420;color:#e95420 !important}.sidebar nav[role=doc-toc] ul>li>a:hover,.sidebar nav[role=doc-toc] ul>li>ul>li>a:hover{color:#e95420 !important}kbd,.kbd{color:#333;background-color:#f8f9fa;border:1px solid;border-radius:5px;border-color:#dee2e6}div.hanging-indent{margin-left:1em;text-indent:-1em}.citation a,.footnote-ref{text-decoration:none}.footnotes ol{padding-left:1em}.tippy-content>*{margin-bottom:.7em}.tippy-content>*:last-child{margin-bottom:0}.table a{word-break:break-word}.table>thead{border-top-width:1px;border-top-color:#dee2e6;border-bottom:1px solid #b3b3b3}.callout{margin-top:1.25rem;margin-bottom:1.25rem;border-radius:.25rem;overflow-wrap:break-word}.callout .callout-title-container{overflow-wrap:anywhere}.callout.callout-style-simple{padding:.4em .7em;border-left:5px solid;border-right:1px solid #dee2e6;border-top:1px solid #dee2e6;border-bottom:1px solid #dee2e6}.callout.callout-style-default{border-left:5px solid;border-right:1px solid #dee2e6;border-top:1px solid #dee2e6;border-bottom:1px solid #dee2e6}.callout .callout-body-container{flex-grow:1}.callout.callout-style-simple .callout-body{font-size:.9rem;font-weight:400}.callout.callout-style-default .callout-body{font-size:.9rem;font-weight:400}.callout.callout-titled .callout-body{margin-top:.2em}.callout:not(.no-icon).callout-titled.callout-style-simple .callout-body{padding-left:1.6em}.callout.callout-titled>.callout-header{padding-top:.2em;margin-bottom:-0.2em}.callout.callout-style-simple>div.callout-header{border-bottom:none;font-size:.9rem;font-weight:600;opacity:75%}.callout.callout-style-default>div.callout-header{border-bottom:none;font-weight:600;opacity:85%;font-size:.9rem;padding-left:.5em;padding-right:.5em}.callout.callout-style-default div.callout-body{padding-left:.5em;padding-right:.5em}.callout.callout-style-default div.callout-body>:first-child{margin-top:.5em}.callout>div.callout-header[data-bs-toggle=collapse]{cursor:pointer}.callout.callout-style-default .callout-header[aria-expanded=false],.callout.callout-style-default .callout-header[aria-expanded=true]{padding-top:0px;margin-bottom:0px;align-items:center}.callout.callout-titled .callout-body>:last-child:not(.sourceCode),.callout.callout-titled .callout-body>div>:last-child:not(.sourceCode){margin-bottom:.5rem}.callout:not(.callout-titled) .callout-body>:first-child,.callout:not(.callout-titled) .callout-body>div>:first-child{margin-top:.25rem}.callout:not(.callout-titled) .callout-body>:last-child,.callout:not(.callout-titled) .callout-body>div>:last-child{margin-bottom:.2rem}.callout.callout-style-simple .callout-icon::before,.callout.callout-style-simple .callout-toggle::before{height:1rem;width:1rem;display:inline-block;content:"";background-repeat:no-repeat;background-size:1rem 1rem}.callout.callout-style-default .callout-icon::before,.callout.callout-style-default .callout-toggle::before{height:.9rem;width:.9rem;display:inline-block;content:"";background-repeat:no-repeat;background-size:.9rem .9rem}.callout.callout-style-default .callout-toggle::before{margin-top:5px}.callout .callout-btn-toggle .callout-toggle::before{transition:transform .2s linear}.callout .callout-header[aria-expanded=false] .callout-toggle::before{transform:rotate(-90deg)}.callout .callout-header[aria-expanded=true] .callout-toggle::before{transform:none}.callout.callout-style-simple:not(.no-icon) div.callout-icon-container{padding-top:.2em;padding-right:.55em}.callout.callout-style-default:not(.no-icon) div.callout-icon-container{padding-top:.1em;padding-right:.35em}.callout.callout-style-default:not(.no-icon) div.callout-title-container{margin-top:-1px}.callout.callout-style-default.callout-caution:not(.no-icon) div.callout-icon-container{padding-top:.3em;padding-right:.35em}.callout>.callout-body>.callout-icon-container>.no-icon,.callout>.callout-header>.callout-icon-container>.no-icon{display:none}div.callout.callout{border-left-color:#6c757d}div.callout.callout-style-default>.callout-header{background-color:#6c757d}div.callout-note.callout{border-left-color:#007bff}div.callout-note.callout-style-default>.callout-header{background-color:#e6f2ff}div.callout-note:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-note.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-note .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-tip.callout{border-left-color:#38b44a}div.callout-tip.callout-style-default>.callout-header{background-color:#ebf8ed}div.callout-tip:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-tip.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-tip .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-warning.callout{border-left-color:#efb73e}div.callout-warning.callout-style-default>.callout-header{background-color:#fdf8ec}div.callout-warning:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-warning.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-warning .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-caution.callout{border-left-color:#e95420}div.callout-caution.callout-style-default>.callout-header{background-color:#fdeee9}div.callout-caution:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-caution.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-caution .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-important.callout{border-left-color:#df382c}div.callout-important.callout-style-default>.callout-header{background-color:#fcebea}div.callout-important:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-important.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-important .callout-toggle::before{background-image:url('data:image/svg+xml,')}.quarto-toggle-container{display:flex;align-items:center}.quarto-reader-toggle .bi::before,.quarto-color-scheme-toggle .bi::before{display:inline-block;height:1rem;width:1rem;content:"";background-repeat:no-repeat;background-size:1rem 1rem}.sidebar-navigation{padding-left:20px}.navbar .quarto-color-scheme-toggle:not(.alternate) .bi::before{background-image:url('data:image/svg+xml,')}.navbar .quarto-color-scheme-toggle.alternate .bi::before{background-image:url('data:image/svg+xml,')}.sidebar-navigation .quarto-color-scheme-toggle:not(.alternate) .bi::before{background-image:url('data:image/svg+xml,')}.sidebar-navigation .quarto-color-scheme-toggle.alternate .bi::before{background-image:url('data:image/svg+xml,')}.quarto-sidebar-toggle{border-color:#dee2e6;border-bottom-left-radius:.25rem;border-bottom-right-radius:.25rem;border-style:solid;border-width:1px;overflow:hidden;border-top-width:0px;padding-top:0px !important}.quarto-sidebar-toggle-title{cursor:pointer;padding-bottom:2px;margin-left:.25em;text-align:center;font-weight:400;font-size:.775em}#quarto-content .quarto-sidebar-toggle{background:#fafafa}#quarto-content .quarto-sidebar-toggle-title{color:#333}.quarto-sidebar-toggle-icon{color:#dee2e6;margin-right:.5em;float:right;transition:transform .2s ease}.quarto-sidebar-toggle-icon::before{padding-top:5px}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-icon{transform:rotate(-180deg)}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-title{border-bottom:solid #dee2e6 1px}.quarto-sidebar-toggle-contents{background-color:#fff;padding-right:10px;padding-left:10px;margin-top:0px !important;transition:max-height .5s ease}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-contents{padding-top:1em;padding-bottom:10px}.quarto-sidebar-toggle:not(.expanded) .quarto-sidebar-toggle-contents{padding-top:0px !important;padding-bottom:0px}nav[role=doc-toc]{z-index:1020}#quarto-sidebar>*,nav[role=doc-toc]>*{transition:opacity .1s ease,border .1s ease}#quarto-sidebar.slow>*,nav[role=doc-toc].slow>*{transition:opacity .4s ease,border .4s ease}.quarto-color-scheme-toggle:not(.alternate).top-right .bi::before{background-image:url('data:image/svg+xml,')}.quarto-color-scheme-toggle.alternate.top-right .bi::before{background-image:url('data:image/svg+xml,')}#quarto-appendix.default{border-top:1px solid #dee2e6}#quarto-appendix.default{background-color:#fff;padding-top:1.5em;margin-top:2em;z-index:998}#quarto-appendix.default .quarto-appendix-heading{margin-top:0;line-height:1.4em;font-weight:600;opacity:.9;border-bottom:none;margin-bottom:0}#quarto-appendix.default .footnotes ol,#quarto-appendix.default .footnotes ol li>p:last-of-type,#quarto-appendix.default .quarto-appendix-contents>p:last-of-type{margin-bottom:0}#quarto-appendix.default .quarto-appendix-secondary-label{margin-bottom:.4em}#quarto-appendix.default .quarto-appendix-bibtex{font-size:.7em;padding:1em;border:solid 1px #dee2e6;margin-bottom:1em}#quarto-appendix.default .quarto-appendix-bibtex code.sourceCode{white-space:pre-wrap}#quarto-appendix.default .quarto-appendix-citeas{font-size:.9em;padding:1em;border:solid 1px #dee2e6;margin-bottom:1em}#quarto-appendix.default .quarto-appendix-heading{font-size:1em !important}#quarto-appendix.default *[role=doc-endnotes]>ol,#quarto-appendix.default .quarto-appendix-contents>*:not(h2):not(.h2){font-size:.9em}#quarto-appendix.default section{padding-bottom:1.5em}#quarto-appendix.default section *[role=doc-endnotes],#quarto-appendix.default section>*:not(a){opacity:.9;word-wrap:break-word}.btn.btn-quarto,div.cell-output-display .btn-quarto{color:#26282a;background-color:#adb5bd;border-color:#adb5bd}.btn.btn-quarto:hover,div.cell-output-display .btn-quarto:hover{color:#26282a;background-color:#b9c0c7;border-color:#b5bcc4}.btn-check:focus+.btn.btn-quarto,.btn.btn-quarto:focus,.btn-check:focus+div.cell-output-display .btn-quarto,div.cell-output-display .btn-quarto:focus{color:#26282a;background-color:#b9c0c7;border-color:#b5bcc4;box-shadow:0 0 0 .25rem rgba(153,160,167,.5)}.btn-check:checked+.btn.btn-quarto,.btn-check:active+.btn.btn-quarto,.btn.btn-quarto:active,.btn.btn-quarto.active,.show>.btn.btn-quarto.dropdown-toggle,.btn-check:checked+div.cell-output-display .btn-quarto,.btn-check:active+div.cell-output-display .btn-quarto,div.cell-output-display .btn-quarto:active,div.cell-output-display .btn-quarto.active,.show>div.cell-output-display .btn-quarto.dropdown-toggle{color:#000;background-color:#bdc4ca;border-color:#b5bcc4}.btn-check:checked+.btn.btn-quarto:focus,.btn-check:active+.btn.btn-quarto:focus,.btn.btn-quarto:active:focus,.btn.btn-quarto.active:focus,.show>.btn.btn-quarto.dropdown-toggle:focus,.btn-check:checked+div.cell-output-display .btn-quarto:focus,.btn-check:active+div.cell-output-display .btn-quarto:focus,div.cell-output-display .btn-quarto:active:focus,div.cell-output-display .btn-quarto.active:focus,.show>div.cell-output-display .btn-quarto.dropdown-toggle:focus{box-shadow:0 0 0 .25rem rgba(153,160,167,.5)}.btn.btn-quarto:disabled,.btn.btn-quarto.disabled,div.cell-output-display .btn-quarto:disabled,div.cell-output-display .btn-quarto.disabled{color:#fff;background-color:#adb5bd;border-color:#adb5bd}nav.quarto-secondary-nav.color-navbar{background-color:#e95420;color:rgba(255,255,255,.8)}nav.quarto-secondary-nav.color-navbar h1,nav.quarto-secondary-nav.color-navbar .h1,nav.quarto-secondary-nav.color-navbar .quarto-btn-toggle{color:rgba(255,255,255,.8)}@media(max-width: 991.98px){body.nav-sidebar .quarto-title-banner{margin-bottom:0;padding-bottom:0}body.nav-sidebar #title-block-header{margin-block-end:0}}p.subtitle{margin-top:.25em;margin-bottom:.5em}code a:any-link{color:inherit;text-decoration-color:#6c757d}/*! light */div.observablehq table thead tr th{background-color:var(--bs-body-bg)}input,button,select,optgroup,textarea{background-color:var(--bs-body-bg)}.code-annotated .code-copy-button{margin-right:1.25em;margin-top:0;padding-bottom:0;padding-top:3px}.code-annotation-gutter-bg{background-color:#fff}.code-annotation-gutter{background-color:rgba(233,236,239,.65)}.code-annotation-gutter,.code-annotation-gutter-bg{height:100%;width:calc(20px + .5em);position:absolute;top:0;right:0}dl.code-annotation-container-grid dt{margin-right:1em;margin-top:.25rem}dl.code-annotation-container-grid dt{font-family:var(--bs-font-monospace);color:#4d4d4d;border:solid #4d4d4d 1px;border-radius:50%;height:22px;width:22px;line-height:22px;font-size:11px;text-align:center;vertical-align:middle;text-decoration:none}dl.code-annotation-container-grid dt[data-target-cell]{cursor:pointer}dl.code-annotation-container-grid dt[data-target-cell].code-annotation-active{color:#fff;border:solid #aaa 1px;background-color:#aaa}pre.code-annotation-code{padding-top:0;padding-bottom:0}pre.code-annotation-code code{z-index:3}#code-annotation-line-highlight-gutter{width:100%;border-top:solid rgba(170,170,170,.2666666667) 1px;border-bottom:solid rgba(170,170,170,.2666666667) 1px;z-index:2;background-color:rgba(170,170,170,.1333333333)}#code-annotation-line-highlight{margin-left:-4em;width:calc(100% + 4em);border-top:solid rgba(170,170,170,.2666666667) 1px;border-bottom:solid rgba(170,170,170,.2666666667) 1px;z-index:2;background-color:rgba(170,170,170,.1333333333)}code.sourceCode .code-annotation-anchor.code-annotation-active{background-color:var(--quarto-hl-normal-color, #aaaaaa);border:solid var(--quarto-hl-normal-color, #aaaaaa) 1px;color:#e9ecef;font-weight:bolder}code.sourceCode .code-annotation-anchor{font-family:var(--bs-font-monospace);color:var(--quarto-hl-co-color);border:solid var(--quarto-hl-co-color) 1px;border-radius:50%;height:18px;width:18px;font-size:9px;margin-top:2px}code.sourceCode button.code-annotation-anchor{padding:2px}code.sourceCode a.code-annotation-anchor{line-height:18px;text-align:center;vertical-align:middle;cursor:default;text-decoration:none}@media print{.page-columns .column-screen-inset{grid-column:page-start-inset/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset table{background:#fff}.page-columns .column-screen-inset-left{grid-column:page-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-left table{background:#fff}.page-columns .column-screen-inset-right{grid-column:body-content-start/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-right table{background:#fff}.page-columns .column-screen{grid-column:page-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen table{background:#fff}.page-columns .column-screen-left{grid-column:page-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-left table{background:#fff}.page-columns .column-screen-right{grid-column:body-content-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-right table{background:#fff}.page-columns .column-screen-inset-shaded{grid-column:page-start-inset/page-end-inset;padding:1em;background:#e9ecef;z-index:998;transform:translate3d(0, 0, 0);margin-bottom:1em}}.quarto-video{margin-bottom:1em}.table>thead{border-top-width:0}.table>:not(caption)>*:not(:last-child)>*{border-bottom-color:#e6e6e6;border-bottom-style:solid;border-bottom-width:1px}.table>:not(:first-child){border-top:1px solid #b3b3b3;border-bottom:1px solid inherit}.table tbody{border-bottom-color:#b3b3b3}a.external:after{display:inline-block;height:.75rem;width:.75rem;margin-bottom:.15em;margin-left:.25em;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:.75rem .75rem}div.sourceCode code a.external:after{content:none}a.external:after:hover{cursor:pointer}.quarto-ext-icon{display:inline-block;font-size:.75em;padding-left:.3em}.code-with-filename .code-with-filename-file{margin-bottom:0;padding-bottom:2px;padding-top:2px;padding-left:.7em;border:var(--quarto-border-width) solid var(--quarto-border-color);border-radius:var(--quarto-border-radius);border-bottom:0;border-bottom-left-radius:0%;border-bottom-right-radius:0%}.code-with-filename div.sourceCode,.reveal .code-with-filename div.sourceCode{margin-top:0;border-top-left-radius:0%;border-top-right-radius:0%}.code-with-filename .code-with-filename-file pre{margin-bottom:0}.code-with-filename .code-with-filename-file,.code-with-filename .code-with-filename-file pre{background-color:rgba(219,219,219,.8)}.quarto-dark .code-with-filename .code-with-filename-file,.quarto-dark .code-with-filename .code-with-filename-file pre{background-color:#555}.code-with-filename .code-with-filename-file strong{font-weight:400}.badge.bg-light{color:#333}.quarto-title-banner{margin-bottom:1em;color:rgba(255,255,255,.8);background:#e95420}.quarto-title-banner .code-tools-button{color:rgba(204,204,204,.8)}.quarto-title-banner .code-tools-button:hover{color:rgba(255,255,255,.8)}.quarto-title-banner .code-tools-button>.bi::before{background-image:url('data:image/svg+xml,')}.quarto-title-banner .code-tools-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}.quarto-title-banner .quarto-title .title{font-weight:600}.quarto-title-banner .quarto-categories{margin-top:.75em}@media(min-width: 992px){.quarto-title-banner{padding-top:2.5em;padding-bottom:2.5em}}@media(max-width: 991.98px){.quarto-title-banner{padding-top:1em;padding-bottom:1em}}main.quarto-banner-title-block>section:first-child>h2,main.quarto-banner-title-block>section:first-child>.h2,main.quarto-banner-title-block>section:first-child>h3,main.quarto-banner-title-block>section:first-child>.h3,main.quarto-banner-title-block>section:first-child>h4,main.quarto-banner-title-block>section:first-child>.h4{margin-top:0}.quarto-title .quarto-categories{display:flex;flex-wrap:wrap;row-gap:.5em;column-gap:.4em;padding-bottom:.5em;margin-top:.75em}.quarto-title .quarto-categories .quarto-category{padding:.25em .75em;font-size:.65em;text-transform:uppercase;border:solid 1px;border-radius:.25rem;opacity:.6}.quarto-title .quarto-categories .quarto-category a{color:inherit}#title-block-header.quarto-title-block.default .quarto-title-meta{display:grid;grid-template-columns:repeat(2, 1fr)}#title-block-header.quarto-title-block.default .quarto-title .title{margin-bottom:0}#title-block-header.quarto-title-block.default .quarto-title-author-orcid img{margin-top:-5px}#title-block-header.quarto-title-block.default .quarto-description p:last-of-type{margin-bottom:0}#title-block-header.quarto-title-block.default .quarto-title-meta-contents p,#title-block-header.quarto-title-block.default .quarto-title-authors p,#title-block-header.quarto-title-block.default .quarto-title-affiliations p{margin-bottom:.1em}#title-block-header.quarto-title-block.default .quarto-title-meta-heading{text-transform:uppercase;margin-top:1em;font-size:.8em;opacity:.8;font-weight:400}#title-block-header.quarto-title-block.default .quarto-title-meta-contents{font-size:.9em}#title-block-header.quarto-title-block.default .quarto-title-meta-contents a{color:#333}#title-block-header.quarto-title-block.default .quarto-title-meta-contents p.affiliation:last-of-type{margin-bottom:.7em}#title-block-header.quarto-title-block.default p.affiliation{margin-bottom:.1em}#title-block-header.quarto-title-block.default .description,#title-block-header.quarto-title-block.default .abstract{margin-top:0}#title-block-header.quarto-title-block.default .description>p,#title-block-header.quarto-title-block.default .abstract>p{font-size:.9em}#title-block-header.quarto-title-block.default .description>p:last-of-type,#title-block-header.quarto-title-block.default .abstract>p:last-of-type{margin-bottom:0}#title-block-header.quarto-title-block.default .description .abstract-title,#title-block-header.quarto-title-block.default .abstract .abstract-title{margin-top:1em;text-transform:uppercase;font-size:.8em;opacity:.8;font-weight:400}#title-block-header.quarto-title-block.default .quarto-title-meta-author{display:grid;grid-template-columns:1fr 1fr}.quarto-title-tools-only{display:flex;justify-content:right}/*# sourceMappingURL=945575463e70190d99eb671cb8520afc.css.map */
diff --git a/site_libs/bootstrap/bootstrap.min.css b/site_libs/bootstrap/bootstrap.min.css
index 938f678..ca57ce6 100644
--- a/site_libs/bootstrap/bootstrap.min.css
+++ b/site_libs/bootstrap/bootstrap.min.css
@@ -7,4 +7,4 @@
*
* ansi colors from IPython notebook's
*
-*/.ansi-black-fg{color:#3e424d}.ansi-black-bg{background-color:#3e424d}.ansi-black-intense-fg{color:#282c36}.ansi-black-intense-bg{background-color:#282c36}.ansi-red-fg{color:#e75c58}.ansi-red-bg{background-color:#e75c58}.ansi-red-intense-fg{color:#b22b31}.ansi-red-intense-bg{background-color:#b22b31}.ansi-green-fg{color:#00a250}.ansi-green-bg{background-color:#00a250}.ansi-green-intense-fg{color:#007427}.ansi-green-intense-bg{background-color:#007427}.ansi-yellow-fg{color:#ddb62b}.ansi-yellow-bg{background-color:#ddb62b}.ansi-yellow-intense-fg{color:#b27d12}.ansi-yellow-intense-bg{background-color:#b27d12}.ansi-blue-fg{color:#208ffb}.ansi-blue-bg{background-color:#208ffb}.ansi-blue-intense-fg{color:#0065ca}.ansi-blue-intense-bg{background-color:#0065ca}.ansi-magenta-fg{color:#d160c4}.ansi-magenta-bg{background-color:#d160c4}.ansi-magenta-intense-fg{color:#a03196}.ansi-magenta-intense-bg{background-color:#a03196}.ansi-cyan-fg{color:#60c6c8}.ansi-cyan-bg{background-color:#60c6c8}.ansi-cyan-intense-fg{color:#258f8f}.ansi-cyan-intense-bg{background-color:#258f8f}.ansi-white-fg{color:#c5c1b4}.ansi-white-bg{background-color:#c5c1b4}.ansi-white-intense-fg{color:#a1a6b2}.ansi-white-intense-bg{background-color:#a1a6b2}.ansi-default-inverse-fg{color:#fff}.ansi-default-inverse-bg{background-color:#000}.ansi-bold{font-weight:bold}.ansi-underline{text-decoration:underline}:root{--quarto-body-bg: #272b30;--quarto-body-color: #adb5bd;--quarto-text-muted: #6c757d;--quarto-border-color: rgba(0, 0, 0, 0.6);--quarto-border-width: 1px;--quarto-border-radius: 0.25rem}table.gt_table{color:var(--quarto-body-color);font-size:1em;width:100%;background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_column_spanner_outer{color:var(--quarto-body-color);background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_col_heading{color:var(--quarto-body-color);font-weight:bold;background-color:rgba(0,0,0,0)}table.gt_table thead.gt_col_headings{border-bottom:1px solid currentColor;border-top-width:inherit;border-top-color:var(--quarto-border-color)}table.gt_table thead.gt_col_headings:not(:first-child){border-top-width:1px;border-top-color:var(--quarto-border-color)}table.gt_table td.gt_row{border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-width:0px}table.gt_table tbody.gt_table_body{border-top-width:1px;border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-color:currentColor}div.columns{display:initial;gap:initial}div.column{display:inline-block;overflow-x:initial;vertical-align:top;width:50%}.code-annotation-tip-content{word-wrap:break-word}.code-annotation-container-hidden{display:none !important}dl.code-annotation-container-grid{display:grid;grid-template-columns:min-content auto}dl.code-annotation-container-grid dt{grid-column:1}dl.code-annotation-container-grid dd{grid-column:2}pre.sourceCode.code-annotation-code{padding-right:0}code.sourceCode .code-annotation-anchor{z-index:100;position:absolute;right:.5em;left:inherit;background-color:rgba(0,0,0,0)}:root{--mermaid-bg-color: #272b30;--mermaid-edge-color: #6c757d;--mermaid-node-fg-color: #adb5bd;--mermaid-fg-color: #adb5bd;--mermaid-fg-color--lighter: #b5bcc4;--mermaid-fg-color--lightest: #bdc4ca;--mermaid-font-family: system-ui, -apple-system, Segoe UI, Roboto, Helvetica Neue, Arial, Noto Sans, Liberation Sans, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji;--mermaid-label-bg-color: #272b30;--mermaid-label-fg-color: #3e444a;--mermaid-node-bg-color: rgba(62, 68, 74, 0.1);--mermaid-node-fg-color: #adb5bd}@media print{:root{font-size:11pt}#quarto-sidebar,#TOC,.nav-page{display:none}.page-columns .content{grid-column-start:page-start}.fixed-top{position:relative}.panel-caption,.figure-caption,figcaption{color:#666}}.code-copy-button{position:absolute;top:0;right:0;border:0;margin-top:5px;margin-right:5px;background-color:rgba(0,0,0,0);z-index:3}.code-copy-button:focus{outline:none}.code-copy-button-tooltip{font-size:.75em}.code-copy-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}.code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}.code-copy-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}.code-copy-button-checked:hover>.bi::before{background-image:url('data:image/svg+xml,')}main ol ol,main ul ul,main ol ul,main ul ol{margin-bottom:1em}ul>li:not(:has(>p))>ul,ol>li:not(:has(>p))>ul,ul>li:not(:has(>p))>ol,ol>li:not(:has(>p))>ol{margin-bottom:0}ul>li:not(:has(>p))>ul>li:has(>p),ol>li:not(:has(>p))>ul>li:has(>p),ul>li:not(:has(>p))>ol>li:has(>p),ol>li:not(:has(>p))>ol>li:has(>p){margin-top:1rem}body{margin:0}main.page-columns>header>h1.title,main.page-columns>header>.title.h1{margin-bottom:0}@media(min-width: 992px){body .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.fullcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] 35px [page-end-inset page-end] 5fr [screen-end-inset] 1.5em}body.slimcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.listing:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 3em [body-end] 50px [body-end-outset] minmax(0px, 250px) [page-end-inset] minmax(50px, 100px) [page-end] 1fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 175px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 175px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] minmax(25px, 50px) [page-start-inset] minmax(50px, 150px) [body-start-outset] minmax(25px, 50px) [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] minmax(25px, 50px) [body-end-outset] minmax(50px, 150px) [page-end-inset] minmax(25px, 50px) [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(50px, 100px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 50px [page-start-inset] minmax(50px, 150px) [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(450px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 50px [page-start-inset] minmax(50px, 150px) [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(450px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(50px, 150px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] minmax(25px, 50px) [page-start-inset] minmax(50px, 150px) [body-start-outset] minmax(25px, 50px) [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] minmax(25px, 50px) [body-end-outset] minmax(50px, 150px) [page-end-inset] minmax(25px, 50px) [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}}@media(max-width: 991.98px){body .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.fullcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.slimcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.listing:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 1250px - 3em )) [body-content-end body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 145px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 145px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1.5em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(75px, 150px) [page-end-inset] 25px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 4fr [screen-end-inset] 1.5em [screen-end]}body.docked.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 4fr [screen-end-inset] 1.5em [screen-end]}body.floating.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(75px, 150px) [page-end-inset] 25px [page-end] 4fr [screen-end-inset] 1.5em [screen-end]}}@media(max-width: 767.98px){body .page-columns,body.fullcontent:not(.floating):not(.docked) .page-columns,body.slimcontent:not(.floating):not(.docked) .page-columns,body.docked .page-columns,body.docked.slimcontent .page-columns,body.docked.fullcontent .page-columns,body.floating .page-columns,body.floating.slimcontent .page-columns,body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}nav[role=doc-toc]{display:none}}body,.page-row-navigation{grid-template-rows:[page-top] max-content [contents-top] max-content [contents-bottom] max-content [page-bottom]}.page-rows-contents{grid-template-rows:[content-top] minmax(max-content, 1fr) [content-bottom] minmax(60px, max-content) [page-bottom]}.page-full{grid-column:screen-start/screen-end !important}.page-columns>*{grid-column:body-content-start/body-content-end}.page-columns.column-page>*{grid-column:page-start/page-end}.page-columns.column-page-left>*{grid-column:page-start/body-content-end}.page-columns.column-page-right>*{grid-column:body-content-start/page-end}.page-rows{grid-auto-rows:auto}.header{grid-column:screen-start/screen-end;grid-row:page-top/contents-top}#quarto-content{padding:0;grid-column:screen-start/screen-end;grid-row:contents-top/contents-bottom}body.floating .sidebar.sidebar-navigation{grid-column:page-start/body-start;grid-row:content-top/page-bottom}body.docked .sidebar.sidebar-navigation{grid-column:screen-start/body-start;grid-row:content-top/page-bottom}.sidebar.toc-left{grid-column:page-start/body-start;grid-row:content-top/page-bottom}.sidebar.margin-sidebar{grid-column:body-end/page-end;grid-row:content-top/page-bottom}.page-columns .content{grid-column:body-content-start/body-content-end;grid-row:content-top/content-bottom;align-content:flex-start}.page-columns .page-navigation{grid-column:body-content-start/body-content-end;grid-row:content-bottom/page-bottom}.page-columns .footer{grid-column:screen-start/screen-end;grid-row:contents-bottom/page-bottom}.page-columns .column-body{grid-column:body-content-start/body-content-end}.page-columns .column-body-fullbleed{grid-column:body-start/body-end}.page-columns .column-body-outset{grid-column:body-start-outset/body-end-outset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset table{background:#272b30}.page-columns .column-body-outset-left{grid-column:body-start-outset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset-left table{background:#272b30}.page-columns .column-body-outset-right{grid-column:body-content-start/body-end-outset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset-right table{background:#272b30}.page-columns .column-page{grid-column:page-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page table{background:#272b30}.page-columns .column-page-inset{grid-column:page-start-inset/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset table{background:#272b30}.page-columns .column-page-inset-left{grid-column:page-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset-left table{background:#272b30}.page-columns .column-page-inset-right{grid-column:body-content-start/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset-right figcaption table{background:#272b30}.page-columns .column-page-left{grid-column:page-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-left table{background:#272b30}.page-columns .column-page-right{grid-column:body-content-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-right figcaption table{background:#272b30}#quarto-content.page-columns #quarto-margin-sidebar,#quarto-content.page-columns #quarto-sidebar{z-index:1}@media(max-width: 991.98px){#quarto-content.page-columns #quarto-margin-sidebar.collapse,#quarto-content.page-columns #quarto-sidebar.collapse,#quarto-content.page-columns #quarto-margin-sidebar.collapsing,#quarto-content.page-columns #quarto-sidebar.collapsing{z-index:1055}}#quarto-content.page-columns main.column-page,#quarto-content.page-columns main.column-page-right,#quarto-content.page-columns main.column-page-left{z-index:0}.page-columns .column-screen-inset{grid-column:screen-start-inset/screen-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset table{background:#272b30}.page-columns .column-screen-inset-left{grid-column:screen-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-left table{background:#272b30}.page-columns .column-screen-inset-right{grid-column:body-content-start/screen-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-right table{background:#272b30}.page-columns .column-screen{grid-column:screen-start/screen-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen table{background:#272b30}.page-columns .column-screen-left{grid-column:screen-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-left table{background:#272b30}.page-columns .column-screen-right{grid-column:body-content-start/screen-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-right table{background:#272b30}.page-columns .column-screen-inset-shaded{grid-column:screen-start/screen-end;padding:1em;background:#d6dade;z-index:998;transform:translate3d(0, 0, 0);margin-bottom:1em}.zindex-content{z-index:998;transform:translate3d(0, 0, 0)}.zindex-modal{z-index:1055;transform:translate3d(0, 0, 0)}.zindex-over-content{z-index:999;transform:translate3d(0, 0, 0)}img.img-fluid.column-screen,img.img-fluid.column-screen-inset-shaded,img.img-fluid.column-screen-inset,img.img-fluid.column-screen-inset-left,img.img-fluid.column-screen-inset-right,img.img-fluid.column-screen-left,img.img-fluid.column-screen-right{width:100%}@media(min-width: 992px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-end/page-end !important;z-index:998}.column-sidebar{grid-column:page-start/body-start !important;z-index:998}.column-leftmargin{grid-column:screen-start-inset/body-start !important;z-index:998}.no-row-height{height:1em;overflow:visible}}@media(max-width: 991.98px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-end/page-end !important;z-index:998}.no-row-height{height:1em;overflow:visible}.page-columns.page-full{overflow:visible}.page-columns.toc-left .margin-caption,.page-columns.toc-left div.aside,.page-columns.toc-left aside,.page-columns.toc-left .column-margin{grid-column:body-content-start/body-content-end !important;z-index:998;transform:translate3d(0, 0, 0)}.page-columns.toc-left .no-row-height{height:initial;overflow:initial}}@media(max-width: 767.98px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-content-start/body-content-end !important;z-index:998;transform:translate3d(0, 0, 0)}.no-row-height{height:initial;overflow:initial}#quarto-margin-sidebar{display:none}#quarto-sidebar-toc-left{display:none}.hidden-sm{display:none}}.panel-grid{display:grid;grid-template-rows:repeat(1, 1fr);grid-template-columns:repeat(24, 1fr);gap:1em}.panel-grid .g-col-1{grid-column:auto/span 1}.panel-grid .g-col-2{grid-column:auto/span 2}.panel-grid .g-col-3{grid-column:auto/span 3}.panel-grid .g-col-4{grid-column:auto/span 4}.panel-grid .g-col-5{grid-column:auto/span 5}.panel-grid .g-col-6{grid-column:auto/span 6}.panel-grid .g-col-7{grid-column:auto/span 7}.panel-grid .g-col-8{grid-column:auto/span 8}.panel-grid .g-col-9{grid-column:auto/span 9}.panel-grid .g-col-10{grid-column:auto/span 10}.panel-grid .g-col-11{grid-column:auto/span 11}.panel-grid .g-col-12{grid-column:auto/span 12}.panel-grid .g-col-13{grid-column:auto/span 13}.panel-grid .g-col-14{grid-column:auto/span 14}.panel-grid .g-col-15{grid-column:auto/span 15}.panel-grid .g-col-16{grid-column:auto/span 16}.panel-grid .g-col-17{grid-column:auto/span 17}.panel-grid .g-col-18{grid-column:auto/span 18}.panel-grid .g-col-19{grid-column:auto/span 19}.panel-grid .g-col-20{grid-column:auto/span 20}.panel-grid .g-col-21{grid-column:auto/span 21}.panel-grid .g-col-22{grid-column:auto/span 22}.panel-grid .g-col-23{grid-column:auto/span 23}.panel-grid .g-col-24{grid-column:auto/span 24}.panel-grid .g-start-1{grid-column-start:1}.panel-grid .g-start-2{grid-column-start:2}.panel-grid .g-start-3{grid-column-start:3}.panel-grid .g-start-4{grid-column-start:4}.panel-grid .g-start-5{grid-column-start:5}.panel-grid .g-start-6{grid-column-start:6}.panel-grid .g-start-7{grid-column-start:7}.panel-grid .g-start-8{grid-column-start:8}.panel-grid .g-start-9{grid-column-start:9}.panel-grid .g-start-10{grid-column-start:10}.panel-grid .g-start-11{grid-column-start:11}.panel-grid .g-start-12{grid-column-start:12}.panel-grid .g-start-13{grid-column-start:13}.panel-grid .g-start-14{grid-column-start:14}.panel-grid .g-start-15{grid-column-start:15}.panel-grid .g-start-16{grid-column-start:16}.panel-grid .g-start-17{grid-column-start:17}.panel-grid .g-start-18{grid-column-start:18}.panel-grid .g-start-19{grid-column-start:19}.panel-grid .g-start-20{grid-column-start:20}.panel-grid .g-start-21{grid-column-start:21}.panel-grid .g-start-22{grid-column-start:22}.panel-grid .g-start-23{grid-column-start:23}@media(min-width: 576px){.panel-grid .g-col-sm-1{grid-column:auto/span 1}.panel-grid .g-col-sm-2{grid-column:auto/span 2}.panel-grid .g-col-sm-3{grid-column:auto/span 3}.panel-grid .g-col-sm-4{grid-column:auto/span 4}.panel-grid .g-col-sm-5{grid-column:auto/span 5}.panel-grid .g-col-sm-6{grid-column:auto/span 6}.panel-grid .g-col-sm-7{grid-column:auto/span 7}.panel-grid .g-col-sm-8{grid-column:auto/span 8}.panel-grid .g-col-sm-9{grid-column:auto/span 9}.panel-grid .g-col-sm-10{grid-column:auto/span 10}.panel-grid .g-col-sm-11{grid-column:auto/span 11}.panel-grid .g-col-sm-12{grid-column:auto/span 12}.panel-grid .g-col-sm-13{grid-column:auto/span 13}.panel-grid .g-col-sm-14{grid-column:auto/span 14}.panel-grid .g-col-sm-15{grid-column:auto/span 15}.panel-grid .g-col-sm-16{grid-column:auto/span 16}.panel-grid .g-col-sm-17{grid-column:auto/span 17}.panel-grid .g-col-sm-18{grid-column:auto/span 18}.panel-grid .g-col-sm-19{grid-column:auto/span 19}.panel-grid .g-col-sm-20{grid-column:auto/span 20}.panel-grid .g-col-sm-21{grid-column:auto/span 21}.panel-grid .g-col-sm-22{grid-column:auto/span 22}.panel-grid .g-col-sm-23{grid-column:auto/span 23}.panel-grid .g-col-sm-24{grid-column:auto/span 24}.panel-grid .g-start-sm-1{grid-column-start:1}.panel-grid .g-start-sm-2{grid-column-start:2}.panel-grid .g-start-sm-3{grid-column-start:3}.panel-grid .g-start-sm-4{grid-column-start:4}.panel-grid .g-start-sm-5{grid-column-start:5}.panel-grid .g-start-sm-6{grid-column-start:6}.panel-grid .g-start-sm-7{grid-column-start:7}.panel-grid .g-start-sm-8{grid-column-start:8}.panel-grid .g-start-sm-9{grid-column-start:9}.panel-grid .g-start-sm-10{grid-column-start:10}.panel-grid .g-start-sm-11{grid-column-start:11}.panel-grid .g-start-sm-12{grid-column-start:12}.panel-grid .g-start-sm-13{grid-column-start:13}.panel-grid .g-start-sm-14{grid-column-start:14}.panel-grid .g-start-sm-15{grid-column-start:15}.panel-grid .g-start-sm-16{grid-column-start:16}.panel-grid .g-start-sm-17{grid-column-start:17}.panel-grid .g-start-sm-18{grid-column-start:18}.panel-grid .g-start-sm-19{grid-column-start:19}.panel-grid .g-start-sm-20{grid-column-start:20}.panel-grid .g-start-sm-21{grid-column-start:21}.panel-grid .g-start-sm-22{grid-column-start:22}.panel-grid .g-start-sm-23{grid-column-start:23}}@media(min-width: 768px){.panel-grid .g-col-md-1{grid-column:auto/span 1}.panel-grid .g-col-md-2{grid-column:auto/span 2}.panel-grid .g-col-md-3{grid-column:auto/span 3}.panel-grid .g-col-md-4{grid-column:auto/span 4}.panel-grid .g-col-md-5{grid-column:auto/span 5}.panel-grid .g-col-md-6{grid-column:auto/span 6}.panel-grid .g-col-md-7{grid-column:auto/span 7}.panel-grid .g-col-md-8{grid-column:auto/span 8}.panel-grid .g-col-md-9{grid-column:auto/span 9}.panel-grid .g-col-md-10{grid-column:auto/span 10}.panel-grid .g-col-md-11{grid-column:auto/span 11}.panel-grid .g-col-md-12{grid-column:auto/span 12}.panel-grid .g-col-md-13{grid-column:auto/span 13}.panel-grid .g-col-md-14{grid-column:auto/span 14}.panel-grid .g-col-md-15{grid-column:auto/span 15}.panel-grid .g-col-md-16{grid-column:auto/span 16}.panel-grid .g-col-md-17{grid-column:auto/span 17}.panel-grid .g-col-md-18{grid-column:auto/span 18}.panel-grid .g-col-md-19{grid-column:auto/span 19}.panel-grid .g-col-md-20{grid-column:auto/span 20}.panel-grid .g-col-md-21{grid-column:auto/span 21}.panel-grid .g-col-md-22{grid-column:auto/span 22}.panel-grid .g-col-md-23{grid-column:auto/span 23}.panel-grid .g-col-md-24{grid-column:auto/span 24}.panel-grid .g-start-md-1{grid-column-start:1}.panel-grid .g-start-md-2{grid-column-start:2}.panel-grid .g-start-md-3{grid-column-start:3}.panel-grid .g-start-md-4{grid-column-start:4}.panel-grid .g-start-md-5{grid-column-start:5}.panel-grid .g-start-md-6{grid-column-start:6}.panel-grid .g-start-md-7{grid-column-start:7}.panel-grid .g-start-md-8{grid-column-start:8}.panel-grid .g-start-md-9{grid-column-start:9}.panel-grid .g-start-md-10{grid-column-start:10}.panel-grid .g-start-md-11{grid-column-start:11}.panel-grid .g-start-md-12{grid-column-start:12}.panel-grid .g-start-md-13{grid-column-start:13}.panel-grid .g-start-md-14{grid-column-start:14}.panel-grid .g-start-md-15{grid-column-start:15}.panel-grid .g-start-md-16{grid-column-start:16}.panel-grid .g-start-md-17{grid-column-start:17}.panel-grid .g-start-md-18{grid-column-start:18}.panel-grid .g-start-md-19{grid-column-start:19}.panel-grid .g-start-md-20{grid-column-start:20}.panel-grid .g-start-md-21{grid-column-start:21}.panel-grid .g-start-md-22{grid-column-start:22}.panel-grid .g-start-md-23{grid-column-start:23}}@media(min-width: 992px){.panel-grid .g-col-lg-1{grid-column:auto/span 1}.panel-grid .g-col-lg-2{grid-column:auto/span 2}.panel-grid .g-col-lg-3{grid-column:auto/span 3}.panel-grid .g-col-lg-4{grid-column:auto/span 4}.panel-grid .g-col-lg-5{grid-column:auto/span 5}.panel-grid .g-col-lg-6{grid-column:auto/span 6}.panel-grid .g-col-lg-7{grid-column:auto/span 7}.panel-grid .g-col-lg-8{grid-column:auto/span 8}.panel-grid .g-col-lg-9{grid-column:auto/span 9}.panel-grid .g-col-lg-10{grid-column:auto/span 10}.panel-grid .g-col-lg-11{grid-column:auto/span 11}.panel-grid .g-col-lg-12{grid-column:auto/span 12}.panel-grid .g-col-lg-13{grid-column:auto/span 13}.panel-grid .g-col-lg-14{grid-column:auto/span 14}.panel-grid .g-col-lg-15{grid-column:auto/span 15}.panel-grid .g-col-lg-16{grid-column:auto/span 16}.panel-grid .g-col-lg-17{grid-column:auto/span 17}.panel-grid .g-col-lg-18{grid-column:auto/span 18}.panel-grid .g-col-lg-19{grid-column:auto/span 19}.panel-grid .g-col-lg-20{grid-column:auto/span 20}.panel-grid .g-col-lg-21{grid-column:auto/span 21}.panel-grid .g-col-lg-22{grid-column:auto/span 22}.panel-grid .g-col-lg-23{grid-column:auto/span 23}.panel-grid .g-col-lg-24{grid-column:auto/span 24}.panel-grid .g-start-lg-1{grid-column-start:1}.panel-grid .g-start-lg-2{grid-column-start:2}.panel-grid .g-start-lg-3{grid-column-start:3}.panel-grid .g-start-lg-4{grid-column-start:4}.panel-grid .g-start-lg-5{grid-column-start:5}.panel-grid .g-start-lg-6{grid-column-start:6}.panel-grid .g-start-lg-7{grid-column-start:7}.panel-grid .g-start-lg-8{grid-column-start:8}.panel-grid .g-start-lg-9{grid-column-start:9}.panel-grid .g-start-lg-10{grid-column-start:10}.panel-grid .g-start-lg-11{grid-column-start:11}.panel-grid .g-start-lg-12{grid-column-start:12}.panel-grid .g-start-lg-13{grid-column-start:13}.panel-grid .g-start-lg-14{grid-column-start:14}.panel-grid .g-start-lg-15{grid-column-start:15}.panel-grid .g-start-lg-16{grid-column-start:16}.panel-grid .g-start-lg-17{grid-column-start:17}.panel-grid .g-start-lg-18{grid-column-start:18}.panel-grid .g-start-lg-19{grid-column-start:19}.panel-grid .g-start-lg-20{grid-column-start:20}.panel-grid .g-start-lg-21{grid-column-start:21}.panel-grid .g-start-lg-22{grid-column-start:22}.panel-grid .g-start-lg-23{grid-column-start:23}}@media(min-width: 1200px){.panel-grid .g-col-xl-1{grid-column:auto/span 1}.panel-grid .g-col-xl-2{grid-column:auto/span 2}.panel-grid .g-col-xl-3{grid-column:auto/span 3}.panel-grid .g-col-xl-4{grid-column:auto/span 4}.panel-grid .g-col-xl-5{grid-column:auto/span 5}.panel-grid .g-col-xl-6{grid-column:auto/span 6}.panel-grid .g-col-xl-7{grid-column:auto/span 7}.panel-grid .g-col-xl-8{grid-column:auto/span 8}.panel-grid .g-col-xl-9{grid-column:auto/span 9}.panel-grid .g-col-xl-10{grid-column:auto/span 10}.panel-grid .g-col-xl-11{grid-column:auto/span 11}.panel-grid .g-col-xl-12{grid-column:auto/span 12}.panel-grid .g-col-xl-13{grid-column:auto/span 13}.panel-grid .g-col-xl-14{grid-column:auto/span 14}.panel-grid .g-col-xl-15{grid-column:auto/span 15}.panel-grid .g-col-xl-16{grid-column:auto/span 16}.panel-grid .g-col-xl-17{grid-column:auto/span 17}.panel-grid .g-col-xl-18{grid-column:auto/span 18}.panel-grid .g-col-xl-19{grid-column:auto/span 19}.panel-grid .g-col-xl-20{grid-column:auto/span 20}.panel-grid .g-col-xl-21{grid-column:auto/span 21}.panel-grid .g-col-xl-22{grid-column:auto/span 22}.panel-grid .g-col-xl-23{grid-column:auto/span 23}.panel-grid .g-col-xl-24{grid-column:auto/span 24}.panel-grid .g-start-xl-1{grid-column-start:1}.panel-grid .g-start-xl-2{grid-column-start:2}.panel-grid .g-start-xl-3{grid-column-start:3}.panel-grid .g-start-xl-4{grid-column-start:4}.panel-grid .g-start-xl-5{grid-column-start:5}.panel-grid .g-start-xl-6{grid-column-start:6}.panel-grid .g-start-xl-7{grid-column-start:7}.panel-grid .g-start-xl-8{grid-column-start:8}.panel-grid .g-start-xl-9{grid-column-start:9}.panel-grid .g-start-xl-10{grid-column-start:10}.panel-grid .g-start-xl-11{grid-column-start:11}.panel-grid .g-start-xl-12{grid-column-start:12}.panel-grid .g-start-xl-13{grid-column-start:13}.panel-grid .g-start-xl-14{grid-column-start:14}.panel-grid .g-start-xl-15{grid-column-start:15}.panel-grid .g-start-xl-16{grid-column-start:16}.panel-grid .g-start-xl-17{grid-column-start:17}.panel-grid .g-start-xl-18{grid-column-start:18}.panel-grid .g-start-xl-19{grid-column-start:19}.panel-grid .g-start-xl-20{grid-column-start:20}.panel-grid .g-start-xl-21{grid-column-start:21}.panel-grid .g-start-xl-22{grid-column-start:22}.panel-grid .g-start-xl-23{grid-column-start:23}}@media(min-width: 1400px){.panel-grid .g-col-xxl-1{grid-column:auto/span 1}.panel-grid .g-col-xxl-2{grid-column:auto/span 2}.panel-grid .g-col-xxl-3{grid-column:auto/span 3}.panel-grid .g-col-xxl-4{grid-column:auto/span 4}.panel-grid .g-col-xxl-5{grid-column:auto/span 5}.panel-grid .g-col-xxl-6{grid-column:auto/span 6}.panel-grid .g-col-xxl-7{grid-column:auto/span 7}.panel-grid .g-col-xxl-8{grid-column:auto/span 8}.panel-grid .g-col-xxl-9{grid-column:auto/span 9}.panel-grid .g-col-xxl-10{grid-column:auto/span 10}.panel-grid .g-col-xxl-11{grid-column:auto/span 11}.panel-grid .g-col-xxl-12{grid-column:auto/span 12}.panel-grid .g-col-xxl-13{grid-column:auto/span 13}.panel-grid .g-col-xxl-14{grid-column:auto/span 14}.panel-grid .g-col-xxl-15{grid-column:auto/span 15}.panel-grid .g-col-xxl-16{grid-column:auto/span 16}.panel-grid .g-col-xxl-17{grid-column:auto/span 17}.panel-grid .g-col-xxl-18{grid-column:auto/span 18}.panel-grid .g-col-xxl-19{grid-column:auto/span 19}.panel-grid .g-col-xxl-20{grid-column:auto/span 20}.panel-grid .g-col-xxl-21{grid-column:auto/span 21}.panel-grid .g-col-xxl-22{grid-column:auto/span 22}.panel-grid .g-col-xxl-23{grid-column:auto/span 23}.panel-grid .g-col-xxl-24{grid-column:auto/span 24}.panel-grid .g-start-xxl-1{grid-column-start:1}.panel-grid .g-start-xxl-2{grid-column-start:2}.panel-grid .g-start-xxl-3{grid-column-start:3}.panel-grid .g-start-xxl-4{grid-column-start:4}.panel-grid .g-start-xxl-5{grid-column-start:5}.panel-grid .g-start-xxl-6{grid-column-start:6}.panel-grid .g-start-xxl-7{grid-column-start:7}.panel-grid .g-start-xxl-8{grid-column-start:8}.panel-grid .g-start-xxl-9{grid-column-start:9}.panel-grid .g-start-xxl-10{grid-column-start:10}.panel-grid .g-start-xxl-11{grid-column-start:11}.panel-grid .g-start-xxl-12{grid-column-start:12}.panel-grid .g-start-xxl-13{grid-column-start:13}.panel-grid .g-start-xxl-14{grid-column-start:14}.panel-grid .g-start-xxl-15{grid-column-start:15}.panel-grid .g-start-xxl-16{grid-column-start:16}.panel-grid .g-start-xxl-17{grid-column-start:17}.panel-grid .g-start-xxl-18{grid-column-start:18}.panel-grid .g-start-xxl-19{grid-column-start:19}.panel-grid .g-start-xxl-20{grid-column-start:20}.panel-grid .g-start-xxl-21{grid-column-start:21}.panel-grid .g-start-xxl-22{grid-column-start:22}.panel-grid .g-start-xxl-23{grid-column-start:23}}main{margin-top:1em;margin-bottom:1em}h1,.h1,h2,.h2{opacity:.9;margin-top:2rem;margin-bottom:1rem;font-weight:600}h1.title,.title.h1{margin-top:0}h2,.h2{border-bottom:1px solid rgba(0,0,0,.6);padding-bottom:.5rem}h3,.h3{font-weight:600}h3,.h3,h4,.h4{opacity:.9;margin-top:1.5rem}h5,.h5,h6,.h6{opacity:.9}.header-section-number{color:#82888e}.nav-link.active .header-section-number{color:inherit}mark,.mark{padding:0em}.panel-caption,caption,.figure-caption{font-size:.9rem}.panel-caption,.figure-caption,figcaption{color:#82888e}.table-caption,caption{color:#adb5bd}.quarto-layout-cell[data-ref-parent] caption{color:#82888e}.column-margin figcaption,.margin-caption,div.aside,aside,.column-margin{color:#82888e;font-size:.825rem}.panel-caption.margin-caption{text-align:inherit}.column-margin.column-container p{margin-bottom:0}.column-margin.column-container>*:not(.collapse){padding-top:.5em;padding-bottom:.5em;display:block}.column-margin.column-container>*.collapse:not(.show){display:none}@media(min-width: 768px){.column-margin.column-container .callout-margin-content:first-child{margin-top:4.5em}.column-margin.column-container .callout-margin-content-simple:first-child{margin-top:3.5em}}.margin-caption>*{padding-top:.5em;padding-bottom:.5em}@media(max-width: 767.98px){.quarto-layout-row{flex-direction:column}}.nav-tabs .nav-item{margin-top:1px;cursor:pointer}.tab-content{margin-top:0px;border-left:rgba(0,0,0,.6) 1px solid;border-right:rgba(0,0,0,.6) 1px solid;border-bottom:rgba(0,0,0,.6) 1px solid;margin-left:0;padding:1em;margin-bottom:1em}@media(max-width: 767.98px){.layout-sidebar{margin-left:0;margin-right:0}}.panel-sidebar,.panel-sidebar .form-control,.panel-input,.panel-input .form-control,.selectize-dropdown{font-size:.9rem}.panel-sidebar .form-control,.panel-input .form-control{padding-top:.1rem}.tab-pane div.sourceCode{margin-top:0px}.tab-pane>p{padding-top:1em}.tab-content>.tab-pane:not(.active){display:none !important}div.sourceCode{background-color:rgba(37,41,46,.65);border:1px solid rgba(37,41,46,.65);border-radius:.25rem}pre.sourceCode{background-color:rgba(0,0,0,0)}pre.sourceCode{border:none;font-size:.875em;overflow:visible !important;padding:.4em}.callout pre.sourceCode{padding-left:0}div.sourceCode{overflow-y:hidden}.callout div.sourceCode{margin-left:initial}.blockquote{font-size:inherit;padding-left:1rem;padding-right:1.5rem;color:#82888e}.blockquote h1:first-child,.blockquote .h1:first-child,.blockquote h2:first-child,.blockquote .h2:first-child,.blockquote h3:first-child,.blockquote .h3:first-child,.blockquote h4:first-child,.blockquote .h4:first-child,.blockquote h5:first-child,.blockquote .h5:first-child{margin-top:0}pre{background-color:initial;padding:initial;border:initial}p code:not(.sourceCode),li code:not(.sourceCode),td code:not(.sourceCode){background-color:#2c3136;padding:.2em}nav p code:not(.sourceCode),nav li code:not(.sourceCode),nav td code:not(.sourceCode){background-color:rgba(0,0,0,0);padding:0}td code:not(.sourceCode){white-space:pre-wrap}#quarto-embedded-source-code-modal>.modal-dialog{max-width:1000px;padding-left:1.75rem;padding-right:1.75rem}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-body{padding:0}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-body div.sourceCode{margin:0;padding:.2rem .2rem;border-radius:0px;border:none}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-header{padding:.7rem}.code-tools-button{font-size:1rem;padding:.15rem .15rem;margin-left:5px;color:#6c757d;background-color:rgba(0,0,0,0);transition:initial;cursor:pointer}.code-tools-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}.code-tools-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}#quarto-embedded-source-code-modal .code-copy-button>.bi::before{background-image:url('data:image/svg+xml,')}#quarto-embedded-source-code-modal .code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}.sidebar{will-change:top;transition:top 200ms linear;position:sticky;overflow-y:auto;padding-top:1.2em;max-height:100vh}.sidebar.toc-left,.sidebar.margin-sidebar{top:0px;padding-top:1em}.sidebar.toc-left>*,.sidebar.margin-sidebar>*{padding-top:.5em}.sidebar.quarto-banner-title-block-sidebar>*{padding-top:1.65em}figure .quarto-notebook-link{margin-top:.5em}.quarto-notebook-link{font-size:.75em;color:#6c757d;margin-bottom:1em;text-decoration:none;display:block}.quarto-notebook-link:hover{text-decoration:underline;color:#fff}.quarto-notebook-link::before{display:inline-block;height:.75rem;width:.75rem;margin-bottom:0em;margin-right:.25em;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:.75rem .75rem}.quarto-alternate-notebooks i.bi,.quarto-alternate-formats i.bi{margin-right:.4em}.quarto-notebook .cell-container{display:flex}.quarto-notebook .cell-container .cell{flex-grow:4}.quarto-notebook .cell-container .cell-decorator{padding-top:1.5em;padding-right:1em;text-align:right}.quarto-notebook h2,.quarto-notebook .h2{border-bottom:none}.sidebar .quarto-alternate-formats a,.sidebar .quarto-alternate-notebooks a{text-decoration:none}.sidebar .quarto-alternate-formats a:hover,.sidebar .quarto-alternate-notebooks a:hover{color:#fff}.sidebar .quarto-alternate-notebooks h2,.sidebar .quarto-alternate-notebooks .h2,.sidebar .quarto-alternate-formats h2,.sidebar .quarto-alternate-formats .h2,.sidebar nav[role=doc-toc]>h2,.sidebar nav[role=doc-toc]>.h2{font-size:.875rem;font-weight:400;margin-bottom:.5rem;margin-top:.3rem;font-family:inherit;border-bottom:0;padding-bottom:0;padding-top:0px}.sidebar .quarto-alternate-notebooks h2,.sidebar .quarto-alternate-notebooks .h2,.sidebar .quarto-alternate-formats h2,.sidebar .quarto-alternate-formats .h2{margin-top:1rem}.sidebar nav[role=doc-toc]>ul a{border-left:1px solid #e9ecef;padding-left:.6rem}.sidebar .quarto-alternate-notebooks h2>ul a,.sidebar .quarto-alternate-notebooks .h2>ul a,.sidebar .quarto-alternate-formats h2>ul a,.sidebar .quarto-alternate-formats .h2>ul a{border-left:none;padding-left:.6rem}.sidebar .quarto-alternate-notebooks ul a:empty,.sidebar .quarto-alternate-formats ul a:empty,.sidebar nav[role=doc-toc]>ul a:empty{display:none}.sidebar .quarto-alternate-notebooks ul,.sidebar .quarto-alternate-formats ul,.sidebar nav[role=doc-toc] ul{padding-left:0;list-style:none;font-size:.875rem;font-weight:300}.sidebar .quarto-alternate-notebooks ul li a,.sidebar .quarto-alternate-formats ul li a,.sidebar nav[role=doc-toc]>ul li a{line-height:1.1rem;padding-bottom:.2rem;padding-top:.2rem;color:inherit}.sidebar nav[role=doc-toc] ul>li>ul>li>a{padding-left:1.2em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>a{padding-left:2.4em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>a{padding-left:3.6em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>ul>li>a{padding-left:4.8em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>ul>li>ul>li>a{padding-left:6em}.sidebar nav[role=doc-toc] ul>li>a.active,.sidebar nav[role=doc-toc] ul>li>ul>li>a.active{border-left:1px solid #fff;color:#fff !important}.sidebar nav[role=doc-toc] ul>li>a:hover,.sidebar nav[role=doc-toc] ul>li>ul>li>a:hover{color:#fff !important}kbd,.kbd{color:#adb5bd;background-color:#f8f9fa;border:1px solid;border-radius:5px;border-color:rgba(0,0,0,.6)}div.hanging-indent{margin-left:1em;text-indent:-1em}.citation a,.footnote-ref{text-decoration:none}.footnotes ol{padding-left:1em}.tippy-content>*{margin-bottom:.7em}.tippy-content>*:last-child{margin-bottom:0}.table a{word-break:break-word}.table>thead{border-top-width:1px;border-top-color:rgba(0,0,0,.6);border-bottom:1px solid #d6dade}.callout{margin-top:1.25rem;margin-bottom:1.25rem;border-radius:.25rem;overflow-wrap:break-word}.callout .callout-title-container{overflow-wrap:anywhere}.callout.callout-style-simple{padding:.4em .7em;border-left:5px solid;border-right:1px solid rgba(0,0,0,.6);border-top:1px solid rgba(0,0,0,.6);border-bottom:1px solid rgba(0,0,0,.6)}.callout.callout-style-default{border-left:5px solid;border-right:1px solid rgba(0,0,0,.6);border-top:1px solid rgba(0,0,0,.6);border-bottom:1px solid rgba(0,0,0,.6)}.callout .callout-body-container{flex-grow:1}.callout.callout-style-simple .callout-body{font-size:.9rem;font-weight:400}.callout.callout-style-default .callout-body{font-size:.9rem;font-weight:400}.callout.callout-titled .callout-body{margin-top:.2em}.callout:not(.no-icon).callout-titled.callout-style-simple .callout-body{padding-left:1.6em}.callout.callout-titled>.callout-header{padding-top:.2em;margin-bottom:-0.2em}.callout.callout-style-simple>div.callout-header{border-bottom:none;font-size:.9rem;font-weight:600;opacity:75%}.callout.callout-style-default>div.callout-header{border-bottom:none;font-weight:600;opacity:85%;font-size:.9rem;padding-left:.5em;padding-right:.5em}.callout.callout-style-default div.callout-body{padding-left:.5em;padding-right:.5em}.callout.callout-style-default div.callout-body>:first-child{margin-top:.5em}.callout>div.callout-header[data-bs-toggle=collapse]{cursor:pointer}.callout.callout-style-default .callout-header[aria-expanded=false],.callout.callout-style-default .callout-header[aria-expanded=true]{padding-top:0px;margin-bottom:0px;align-items:center}.callout.callout-titled .callout-body>:last-child:not(.sourceCode),.callout.callout-titled .callout-body>div>:last-child:not(.sourceCode){margin-bottom:.5rem}.callout:not(.callout-titled) .callout-body>:first-child,.callout:not(.callout-titled) .callout-body>div>:first-child{margin-top:.25rem}.callout:not(.callout-titled) .callout-body>:last-child,.callout:not(.callout-titled) .callout-body>div>:last-child{margin-bottom:.2rem}.callout.callout-style-simple .callout-icon::before,.callout.callout-style-simple .callout-toggle::before{height:1rem;width:1rem;display:inline-block;content:"";background-repeat:no-repeat;background-size:1rem 1rem}.callout.callout-style-default .callout-icon::before,.callout.callout-style-default .callout-toggle::before{height:.9rem;width:.9rem;display:inline-block;content:"";background-repeat:no-repeat;background-size:.9rem .9rem}.callout.callout-style-default .callout-toggle::before{margin-top:5px}.callout .callout-btn-toggle .callout-toggle::before{transition:transform .2s linear}.callout .callout-header[aria-expanded=false] .callout-toggle::before{transform:rotate(-90deg)}.callout .callout-header[aria-expanded=true] .callout-toggle::before{transform:none}.callout.callout-style-simple:not(.no-icon) div.callout-icon-container{padding-top:.2em;padding-right:.55em}.callout.callout-style-default:not(.no-icon) div.callout-icon-container{padding-top:.1em;padding-right:.35em}.callout.callout-style-default:not(.no-icon) div.callout-title-container{margin-top:-1px}.callout.callout-style-default.callout-caution:not(.no-icon) div.callout-icon-container{padding-top:.3em;padding-right:.35em}.callout>.callout-body>.callout-icon-container>.no-icon,.callout>.callout-header>.callout-icon-container>.no-icon{display:none}div.callout.callout{border-left-color:#6c757d}div.callout.callout-style-default>.callout-header{background-color:#6c757d}div.callout-note.callout{border-left-color:#007bff}div.callout-note.callout-style-default>.callout-header{background-color:#00254d}div.callout-note:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-note.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-note .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-tip.callout{border-left-color:#62c462}div.callout-tip.callout-style-default>.callout-header{background-color:#1d3b1d}div.callout-tip:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-tip.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-tip .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-warning.callout{border-left-color:#f89406}div.callout-warning.callout-style-default>.callout-header{background-color:#4a2c02}div.callout-warning:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-warning.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-warning .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-caution.callout{border-left-color:#fd7e14}div.callout-caution.callout-style-default>.callout-header{background-color:#4c2606}div.callout-caution:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-caution.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-caution .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-important.callout{border-left-color:#ee5f5b}div.callout-important.callout-style-default>.callout-header{background-color:#471d1b}div.callout-important:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-important.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-important .callout-toggle::before{background-image:url('data:image/svg+xml,')}.quarto-toggle-container{display:flex;align-items:center}.quarto-reader-toggle .bi::before,.quarto-color-scheme-toggle .bi::before{display:inline-block;height:1rem;width:1rem;content:"";background-repeat:no-repeat;background-size:1rem 1rem}.sidebar-navigation{padding-left:20px}.navbar .quarto-color-scheme-toggle:not(.alternate) .bi::before{background-image:url('data:image/svg+xml,')}.navbar .quarto-color-scheme-toggle.alternate .bi::before{background-image:url('data:image/svg+xml,')}.sidebar-navigation .quarto-color-scheme-toggle:not(.alternate) .bi::before{background-image:url('data:image/svg+xml,')}.sidebar-navigation .quarto-color-scheme-toggle.alternate .bi::before{background-image:url('data:image/svg+xml,')}.quarto-sidebar-toggle{border-color:#dee2e6;border-bottom-left-radius:.25rem;border-bottom-right-radius:.25rem;border-style:solid;border-width:1px;overflow:hidden;border-top-width:0px;padding-top:0px !important}.quarto-sidebar-toggle-title{cursor:pointer;padding-bottom:2px;margin-left:.25em;text-align:center;font-weight:400;font-size:.775em}#quarto-content .quarto-sidebar-toggle{background:#2b2f34}#quarto-content .quarto-sidebar-toggle-title{color:#adb5bd}.quarto-sidebar-toggle-icon{color:#dee2e6;margin-right:.5em;float:right;transition:transform .2s ease}.quarto-sidebar-toggle-icon::before{padding-top:5px}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-icon{transform:rotate(-180deg)}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-title{border-bottom:solid #dee2e6 1px}.quarto-sidebar-toggle-contents{background-color:#272b30;padding-right:10px;padding-left:10px;margin-top:0px !important;transition:max-height .5s ease}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-contents{padding-top:1em;padding-bottom:10px}.quarto-sidebar-toggle:not(.expanded) .quarto-sidebar-toggle-contents{padding-top:0px !important;padding-bottom:0px}nav[role=doc-toc]{z-index:1020}#quarto-sidebar>*,nav[role=doc-toc]>*{transition:opacity .1s ease,border .1s ease}#quarto-sidebar.slow>*,nav[role=doc-toc].slow>*{transition:opacity .4s ease,border .4s ease}.quarto-color-scheme-toggle:not(.alternate).top-right .bi::before{background-image:url('data:image/svg+xml,')}.quarto-color-scheme-toggle.alternate.top-right .bi::before{background-image:url('data:image/svg+xml,')}#quarto-appendix.default{border-top:1px solid #dee2e6}#quarto-appendix.default{background-color:#272b30;padding-top:1.5em;margin-top:2em;z-index:998}#quarto-appendix.default .quarto-appendix-heading{margin-top:0;line-height:1.4em;font-weight:600;opacity:.9;border-bottom:none;margin-bottom:0}#quarto-appendix.default .footnotes ol,#quarto-appendix.default .footnotes ol li>p:last-of-type,#quarto-appendix.default .quarto-appendix-contents>p:last-of-type{margin-bottom:0}#quarto-appendix.default .quarto-appendix-secondary-label{margin-bottom:.4em}#quarto-appendix.default .quarto-appendix-bibtex{font-size:.7em;padding:1em;border:solid 1px #dee2e6;margin-bottom:1em}#quarto-appendix.default .quarto-appendix-bibtex code.sourceCode{white-space:pre-wrap}#quarto-appendix.default .quarto-appendix-citeas{font-size:.9em;padding:1em;border:solid 1px #dee2e6;margin-bottom:1em}#quarto-appendix.default .quarto-appendix-heading{font-size:1em !important}#quarto-appendix.default *[role=doc-endnotes]>ol,#quarto-appendix.default .quarto-appendix-contents>*:not(h2):not(.h2){font-size:.9em}#quarto-appendix.default section{padding-bottom:1.5em}#quarto-appendix.default section *[role=doc-endnotes],#quarto-appendix.default section>*:not(a){opacity:.9;word-wrap:break-word}.btn.btn-quarto,div.cell-output-display .btn-quarto{color:#fefefe;background-color:#6c757d;background-image:var(--bs-gradient);border-color:#6c757d;box-shadow:inset 0 1px 0 rgba(255,255,255,.15),0 1px 1px rgba(0,0,0,.075)}.btn.btn-quarto:hover,div.cell-output-display .btn-quarto:hover{color:#fefefe;background-color:#828a91;background-image:var(--bs-gradient);border-color:#7b838a}.btn-check:focus+.btn.btn-quarto,.btn.btn-quarto:focus,.btn-check:focus+div.cell-output-display .btn-quarto,div.cell-output-display .btn-quarto:focus{color:#fefefe;background-color:#828a91;background-image:var(--bs-gradient);border-color:#7b838a;box-shadow:inset 0 1px 0 rgba(255,255,255,.15),0 1px 1px rgba(0,0,0,.075),0 0 0 .25rem rgba(130,138,144,.5)}.btn-check:checked+.btn.btn-quarto,.btn-check:active+.btn.btn-quarto,.btn.btn-quarto:active,.btn.btn-quarto.active,.show>.btn.btn-quarto.dropdown-toggle,.btn-check:checked+div.cell-output-display .btn-quarto,.btn-check:active+div.cell-output-display .btn-quarto,div.cell-output-display .btn-quarto:active,div.cell-output-display .btn-quarto.active,.show>div.cell-output-display .btn-quarto.dropdown-toggle{color:#fff;background-color:#899197;background-image:none;border-color:#7b838a}.btn-check:checked+.btn.btn-quarto:focus,.btn-check:active+.btn.btn-quarto:focus,.btn.btn-quarto:active:focus,.btn.btn-quarto.active:focus,.show>.btn.btn-quarto.dropdown-toggle:focus,.btn-check:checked+div.cell-output-display .btn-quarto:focus,.btn-check:active+div.cell-output-display .btn-quarto:focus,div.cell-output-display .btn-quarto:active:focus,div.cell-output-display .btn-quarto.active:focus,.show>div.cell-output-display .btn-quarto.dropdown-toggle:focus{box-shadow:inset 0 3px 5px rgba(0,0,0,.125),0 0 0 .25rem rgba(130,138,144,.5)}.btn.btn-quarto:disabled,.btn.btn-quarto.disabled,div.cell-output-display .btn-quarto:disabled,div.cell-output-display .btn-quarto.disabled{color:#fff;background-color:#6c757d;background-image:none;border-color:#6c757d}nav.quarto-secondary-nav.color-navbar{background-color:#3e444a;color:#d8dadb}nav.quarto-secondary-nav.color-navbar h1,nav.quarto-secondary-nav.color-navbar .h1,nav.quarto-secondary-nav.color-navbar .quarto-btn-toggle{color:#d8dadb}@media(max-width: 991.98px){body.nav-sidebar .quarto-title-banner{margin-bottom:0;padding-bottom:0}body.nav-sidebar #title-block-header{margin-block-end:0}}p.subtitle{margin-top:.25em;margin-bottom:.5em}code a:any-link{color:inherit;text-decoration-color:#6c757d}/*! dark */div.observablehq table thead tr th{background-color:var(--bs-body-bg)}input,button,select,optgroup,textarea{background-color:var(--bs-body-bg)}.code-annotated .code-copy-button{margin-right:1.25em;margin-top:0;padding-bottom:0;padding-top:3px}.code-annotation-gutter-bg{background-color:#272b30}.code-annotation-gutter{background-color:rgba(37,41,46,.65)}.code-annotation-gutter,.code-annotation-gutter-bg{height:100%;width:calc(20px + .5em);position:absolute;top:0;right:0}dl.code-annotation-container-grid dt{margin-right:1em;margin-top:.25rem}dl.code-annotation-container-grid dt{font-family:var(--bs-font-monospace);color:#9ca3aa;border:solid #9ca3aa 1px;border-radius:50%;height:22px;width:22px;line-height:22px;font-size:11px;text-align:center;vertical-align:middle;text-decoration:none}dl.code-annotation-container-grid dt[data-target-cell]{cursor:pointer}dl.code-annotation-container-grid dt[data-target-cell].code-annotation-active{color:#272b30;border:solid #aaa 1px;background-color:#aaa}pre.code-annotation-code{padding-top:0;padding-bottom:0}pre.code-annotation-code code{z-index:3}#code-annotation-line-highlight-gutter{width:100%;border-top:solid rgba(170,170,170,.2666666667) 1px;border-bottom:solid rgba(170,170,170,.2666666667) 1px;z-index:2;background-color:rgba(170,170,170,.1333333333)}#code-annotation-line-highlight{margin-left:-4em;width:calc(100% + 4em);border-top:solid rgba(170,170,170,.2666666667) 1px;border-bottom:solid rgba(170,170,170,.2666666667) 1px;z-index:2;background-color:rgba(170,170,170,.1333333333)}code.sourceCode .code-annotation-anchor.code-annotation-active{background-color:var(--quarto-hl-normal-color, #aaaaaa);border:solid var(--quarto-hl-normal-color, #aaaaaa) 1px;color:#25292e;font-weight:bolder}code.sourceCode .code-annotation-anchor{font-family:var(--bs-font-monospace);color:var(--quarto-hl-co-color);border:solid var(--quarto-hl-co-color) 1px;border-radius:50%;height:18px;width:18px;font-size:9px;margin-top:2px}code.sourceCode button.code-annotation-anchor{padding:2px}code.sourceCode a.code-annotation-anchor{line-height:18px;text-align:center;vertical-align:middle;cursor:default;text-decoration:none}@media print{.page-columns .column-screen-inset{grid-column:page-start-inset/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset table{background:#272b30}.page-columns .column-screen-inset-left{grid-column:page-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-left table{background:#272b30}.page-columns .column-screen-inset-right{grid-column:body-content-start/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-right table{background:#272b30}.page-columns .column-screen{grid-column:page-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen table{background:#272b30}.page-columns .column-screen-left{grid-column:page-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-left table{background:#272b30}.page-columns .column-screen-right{grid-column:body-content-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-right table{background:#272b30}.page-columns .column-screen-inset-shaded{grid-column:page-start-inset/page-end-inset;padding:1em;background:#d6dade;z-index:998;transform:translate3d(0, 0, 0);margin-bottom:1em}}.quarto-video{margin-bottom:1em}.table>thead{border-top-width:0}.table>:not(caption)>*:not(:last-child)>*{border-bottom-color:#e6e9eb;border-bottom-style:solid;border-bottom-width:1px}.table>:not(:first-child){border-top:1px solid #d6dade;border-bottom:1px solid inherit}.table tbody{border-bottom-color:#d6dade}a.external:after{display:inline-block;height:.75rem;width:.75rem;margin-bottom:.15em;margin-left:.25em;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:.75rem .75rem}div.sourceCode code a.external:after{content:none}a.external:after:hover{cursor:pointer}.quarto-ext-icon{display:inline-block;font-size:.75em;padding-left:.3em}.code-with-filename .code-with-filename-file{margin-bottom:0;padding-bottom:2px;padding-top:2px;padding-left:.7em;border:var(--quarto-border-width) solid var(--quarto-border-color);border-radius:var(--quarto-border-radius);border-bottom:0;border-bottom-left-radius:0%;border-bottom-right-radius:0%}.code-with-filename div.sourceCode,.reveal .code-with-filename div.sourceCode{margin-top:0;border-top-left-radius:0%;border-top-right-radius:0%}.code-with-filename .code-with-filename-file pre{margin-bottom:0}.code-with-filename .code-with-filename-file,.code-with-filename .code-with-filename-file pre{background-color:rgba(219,219,219,.8)}.quarto-dark .code-with-filename .code-with-filename-file,.quarto-dark .code-with-filename .code-with-filename-file pre{background-color:#555}.code-with-filename .code-with-filename-file strong{font-weight:400}.quarto-title-banner{margin-bottom:1em;color:#d8dadb;background:#3e444a}.quarto-title-banner .code-tools-button{color:#adaeaf}.quarto-title-banner .code-tools-button:hover{color:#d8dadb}.quarto-title-banner .code-tools-button>.bi::before{background-image:url('data:image/svg+xml,')}.quarto-title-banner .code-tools-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}.quarto-title-banner .quarto-title .title{font-weight:600}.quarto-title-banner .quarto-categories{margin-top:.75em}@media(min-width: 992px){.quarto-title-banner{padding-top:2.5em;padding-bottom:2.5em}}@media(max-width: 991.98px){.quarto-title-banner{padding-top:1em;padding-bottom:1em}}main.quarto-banner-title-block>section:first-child>h2,main.quarto-banner-title-block>section:first-child>.h2,main.quarto-banner-title-block>section:first-child>h3,main.quarto-banner-title-block>section:first-child>.h3,main.quarto-banner-title-block>section:first-child>h4,main.quarto-banner-title-block>section:first-child>.h4{margin-top:0}.quarto-title .quarto-categories{display:flex;flex-wrap:wrap;row-gap:.5em;column-gap:.4em;padding-bottom:.5em;margin-top:.75em}.quarto-title .quarto-categories .quarto-category{padding:.25em .75em;font-size:.65em;text-transform:uppercase;border:solid 1px;border-radius:.25rem;opacity:.6}.quarto-title .quarto-categories .quarto-category a{color:inherit}#title-block-header.quarto-title-block.default .quarto-title-meta{display:grid;grid-template-columns:repeat(2, 1fr)}#title-block-header.quarto-title-block.default .quarto-title .title{margin-bottom:0}#title-block-header.quarto-title-block.default .quarto-title-author-orcid img{margin-top:-5px}#title-block-header.quarto-title-block.default .quarto-description p:last-of-type{margin-bottom:0}#title-block-header.quarto-title-block.default .quarto-title-meta-contents p,#title-block-header.quarto-title-block.default .quarto-title-authors p,#title-block-header.quarto-title-block.default .quarto-title-affiliations p{margin-bottom:.1em}#title-block-header.quarto-title-block.default .quarto-title-meta-heading{text-transform:uppercase;margin-top:1em;font-size:.8em;opacity:.8;font-weight:400}#title-block-header.quarto-title-block.default .quarto-title-meta-contents{font-size:.9em}#title-block-header.quarto-title-block.default .quarto-title-meta-contents a{color:#adb5bd}#title-block-header.quarto-title-block.default .quarto-title-meta-contents p.affiliation:last-of-type{margin-bottom:.7em}#title-block-header.quarto-title-block.default p.affiliation{margin-bottom:.1em}#title-block-header.quarto-title-block.default .description,#title-block-header.quarto-title-block.default .abstract{margin-top:0}#title-block-header.quarto-title-block.default .description>p,#title-block-header.quarto-title-block.default .abstract>p{font-size:.9em}#title-block-header.quarto-title-block.default .description>p:last-of-type,#title-block-header.quarto-title-block.default .abstract>p:last-of-type{margin-bottom:0}#title-block-header.quarto-title-block.default .description .abstract-title,#title-block-header.quarto-title-block.default .abstract .abstract-title{margin-top:1em;text-transform:uppercase;font-size:.8em;opacity:.8;font-weight:400}#title-block-header.quarto-title-block.default .quarto-title-meta-author{display:grid;grid-template-columns:1fr 1fr}.quarto-title-tools-only{display:flex;justify-content:right}.navbar{border:1px solid rgba(0,0,0,.6);text-shadow:1px 1px 1px rgba(0,0,0,.3)}.navbar .container{padding:0}.navbar .navbar-toggler{border-color:rgba(0,0,0,.6)}.navbar-fixed-top{border-width:0 0 1px}.navbar-fixed-bottom{border-width:1px 0 0}.navbar .nav-link{padding:1rem;border-left:1px solid rgba(255,255,255,.1);border-right:1px solid rgba(0,0,0,.2)}.navbar .nav-link:hover,.navbar .nav-link:focus{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;border-left:1px solid rgba(0,0,0,.2)}.navbar-brand{padding:.75rem 1rem calc(24px - 0.75rem);margin-right:0;border-right:1px solid rgba(0,0,0,.2)}.navbar .nav-item.active .nav-link{background-color:rgba(0,0,0,.3);border-left:1px solid rgba(0,0,0,.2)}.navbar-nav .nav-item+.nav-item{margin-left:0}.navbar.bg-light{text-shadow:1px 1px 1px rgba(0,0,0,.1)}.navbar.bg-light .nav-link:hover,.navbar.bg-light .nav-link:focus{background-image:linear-gradient(#6f7479, #797f84 40%, #80868c);filter:none;border-left:1px solid rgba(0,0,0,.2)}@media(max-width: 576px){.navbar-expand-sm .navbar-brand,.navbar-expand-sm .nav-link{border:none !important}}@media(max-width: 768px){.navbar-expand-md .navbar-brand,.navbar-expand-md .nav-link{border:none !important}}@media(max-width: 992px){.navbar-expand-lg .navbar-brand,.navbar-expand-lg .nav-link{border:none !important}}.btn{border-color:rgba(0,0,0,.6);text-shadow:1px 1px 1px rgba(0,0,0,.3)}.btn:not([disabled]):not(.disabled).active,.btn.disabled{border-color:rgba(0,0,0,.6);box-shadow:none}.btn:hover,.btn:focus,.btn:not([disabled]):not(.disabled):active,.btn:not([disabled]):not(.disabled):active:hover,.btn:not([disabled]):not(.disabled).active:hover{border-color:rgba(0,0,0,.6)}.btn-primary{background-image:linear-gradient(#555a60, #3e444a 60%, #393f44);filter:none}.btn-primary:not([disabled]):not(.disabled):hover,.btn-primary:not([disabled]):not(.disabled):focus,.btn-primary:not([disabled]):not(.disabled):active:hover,.btn-primary:not([disabled]):not(.disabled).active:hover{background-image:linear-gradient(#282c2f, #2b3034 40%, #2e3237);filter:none}.btn-default{background-image:linear-gradient(#7e868d, #6c757d 60%, #636c73);filter:none;color:#fff}.btn-default:not([disabled]):not(.disabled):hover,.btn-default:not([disabled]):not(.disabled):focus,.btn-default:not([disabled]):not(.disabled):active,.btn-default:not([disabled]):not(.disabled).active{background-image:linear-gradient(#454b50, #4c5258 40%, #50575d);filter:none}.btn-secondary{background-image:linear-gradient(#7e868d, #6c757d 60%, #636c73);filter:none}.btn-secondary:not([disabled]):not(.disabled):hover,.btn-secondary:not([disabled]):not(.disabled):focus,.btn-secondary:not([disabled]):not(.disabled):active,.btn-secondary:not([disabled]):not(.disabled).active{background-image:linear-gradient(#454b50, #4c5258 40%, #50575d);filter:none}.btn-success{background-image:linear-gradient(#75cb75, #62c462 60%, #5ab45a);filter:none;color:#fff}.btn-success:not([disabled]):not(.disabled):hover,.btn-success:not([disabled]):not(.disabled):focus,.btn-success:not([disabled]):not(.disabled):active,.btn-success:not([disabled]):not(.disabled).active{background-image:linear-gradient(#3f7d3f, #458945 40%, #499149);filter:none}.btn-info{background-image:linear-gradient(#6fc8e2, #5bc0de 60%, #54b1cc);filter:none;color:#fff}.btn-info:not([disabled]):not(.disabled):hover,.btn-info:not([disabled]):not(.disabled):focus,.btn-info:not([disabled]):not(.disabled):active,.btn-info:not([disabled]):not(.disabled).active{background-image:linear-gradient(#3a7b8e, #40869b 40%, #438ea4);filter:none}.btn-warning{background-image:linear-gradient(#f9a124, #f89406 60%, #e48806);filter:none;color:#fff}.btn-warning:not([disabled]):not(.disabled):hover,.btn-warning:not([disabled]):not(.disabled):focus,.btn-warning:not([disabled]):not(.disabled):active,.btn-warning:not([disabled]):not(.disabled).active{background-image:linear-gradient(#9f5f04, #ae6804 40%, #b86e04);filter:none}.btn-danger{background-image:linear-gradient(#f0726f, #ee5f5b 60%, #db5754);filter:none}.btn-danger:not([disabled]):not(.disabled):hover,.btn-danger:not([disabled]):not(.disabled):focus,.btn-danger:not([disabled]):not(.disabled):active,.btn-danger:not([disabled]):not(.disabled).active{background-image:linear-gradient(#983d3a, #a74340 40%, #b04643);filter:none}.btn-outline-primary{color:#fff}.btn-link,.btn-link:hover{border-color:rgba(0,0,0,0)}.btn-group .btn.active,.btn-group-vertical .btn.active{border-color:rgba(0,0,0,.6)}h1,.h1,h2,.h2,h3,.h3,h4,.h4,h5,.h5,h6,.h6{text-shadow:-1px -1px 0 rgba(0,0,0,.3)}legend{color:#fff}.input-group-addon{background-image:linear-gradient(#7e868d, #6c757d 60%, #636c73);filter:none;text-shadow:1px 1px 1px rgba(0,0,0,.3);color:#fff}.nav-tabs .nav-link{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;border:1px solid rgba(0,0,0,.6)}.nav-tabs .nav-link:not([disabled]):not(.disabled):hover,.nav-tabs .nav-link:not([disabled]):not(.disabled):focus,.nav-tabs .nav-link:not([disabled]):not(.disabled):active,.nav-tabs .nav-link:not([disabled]):not(.disabled).active{background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none}.nav-tabs .nav-link.disabled{border:1px solid rgba(0,0,0,.6)}.nav-tabs .nav-link,.nav-tabs .nav-link:hover{color:#fff}.nav-pills .nav-link{background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none;border:1px solid rgba(0,0,0,.6);text-shadow:1px 1px 1px rgba(0,0,0,.3);color:#fff}.nav-pills .nav-link:hover{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;border:1px solid rgba(0,0,0,.6)}.nav-pills .nav-link.active,.nav-pills .nav-link:hover{background-color:rgba(0,0,0,0);background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;border:1px solid rgba(0,0,0,.6)}.nav-pills .nav-link.disabled,.nav-pills .nav-link.disabled:hover{background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none;color:#6c757d}.pagination .page-link{text-shadow:1px 1px 1px rgba(0,0,0,.3);background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none}.pagination .page-link:hover{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;text-decoration:none}.pagination .page-item.active .page-link{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none}.pagination .page-item.disabled .page-link{background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none}.breadcrumb{border:1px solid rgba(0,0,0,.6);text-shadow:1px 1px 1px rgba(0,0,0,.3);background-color:rgba(0,0,0,0);background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none}.breadcrumb a,.breadcrumb a:hover{color:#fff}.alert{border:none;color:#fff}.alert a,.alert .alert-link{color:#fff;text-decoration:underline}.alert-default{background-color:#6c757d}.alert-primary{background-color:#3e444a}.alert-secondary{background-color:#6c757d}.alert-success{background-color:#62c462}.alert-info{background-color:#5bc0de}.alert-warning{background-color:#f89406}.alert-danger{background-color:#ee5f5b}.alert-light{background-color:#d6dade}.alert-dark{background-color:#272b30}.alert-light,.alert-light a:not(.btn),.alert-light .alert-link{color:#272b30}.badge.bg-light{color:#272b30}.list-group-item-action:hover{background-color:#23272b}/*# sourceMappingURL=5045415e7f422438c8b3a791d2c1c7d3.css.map */
+*/.ansi-black-fg{color:#3e424d}.ansi-black-bg{background-color:#3e424d}.ansi-black-intense-fg{color:#282c36}.ansi-black-intense-bg{background-color:#282c36}.ansi-red-fg{color:#e75c58}.ansi-red-bg{background-color:#e75c58}.ansi-red-intense-fg{color:#b22b31}.ansi-red-intense-bg{background-color:#b22b31}.ansi-green-fg{color:#00a250}.ansi-green-bg{background-color:#00a250}.ansi-green-intense-fg{color:#007427}.ansi-green-intense-bg{background-color:#007427}.ansi-yellow-fg{color:#ddb62b}.ansi-yellow-bg{background-color:#ddb62b}.ansi-yellow-intense-fg{color:#b27d12}.ansi-yellow-intense-bg{background-color:#b27d12}.ansi-blue-fg{color:#208ffb}.ansi-blue-bg{background-color:#208ffb}.ansi-blue-intense-fg{color:#0065ca}.ansi-blue-intense-bg{background-color:#0065ca}.ansi-magenta-fg{color:#d160c4}.ansi-magenta-bg{background-color:#d160c4}.ansi-magenta-intense-fg{color:#a03196}.ansi-magenta-intense-bg{background-color:#a03196}.ansi-cyan-fg{color:#60c6c8}.ansi-cyan-bg{background-color:#60c6c8}.ansi-cyan-intense-fg{color:#258f8f}.ansi-cyan-intense-bg{background-color:#258f8f}.ansi-white-fg{color:#c5c1b4}.ansi-white-bg{background-color:#c5c1b4}.ansi-white-intense-fg{color:#a1a6b2}.ansi-white-intense-bg{background-color:#a1a6b2}.ansi-default-inverse-fg{color:#fff}.ansi-default-inverse-bg{background-color:#000}.ansi-bold{font-weight:bold}.ansi-underline{text-decoration:underline}:root{--quarto-body-bg: #272b30;--quarto-body-color: #adb5bd;--quarto-text-muted: #6c757d;--quarto-border-color: rgba(0, 0, 0, 0.6);--quarto-border-width: 1px;--quarto-border-radius: 0.25rem}table.gt_table{color:var(--quarto-body-color);font-size:1em;width:100%;background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_column_spanner_outer{color:var(--quarto-body-color);background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_col_heading{color:var(--quarto-body-color);font-weight:bold;background-color:rgba(0,0,0,0)}table.gt_table thead.gt_col_headings{border-bottom:1px solid currentColor;border-top-width:inherit;border-top-color:var(--quarto-border-color)}table.gt_table thead.gt_col_headings:not(:first-child){border-top-width:1px;border-top-color:var(--quarto-border-color)}table.gt_table td.gt_row{border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-width:0px}table.gt_table tbody.gt_table_body{border-top-width:1px;border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-color:currentColor}div.columns{display:initial;gap:initial}div.column{display:inline-block;overflow-x:initial;vertical-align:top;width:50%}.code-annotation-tip-content{word-wrap:break-word}.code-annotation-container-hidden{display:none !important}dl.code-annotation-container-grid{display:grid;grid-template-columns:min-content auto}dl.code-annotation-container-grid dt{grid-column:1}dl.code-annotation-container-grid dd{grid-column:2}pre.sourceCode.code-annotation-code{padding-right:0}code.sourceCode .code-annotation-anchor{z-index:100;position:absolute;right:.5em;left:inherit;background-color:rgba(0,0,0,0)}:root{--mermaid-bg-color: #272b30;--mermaid-edge-color: #6c757d;--mermaid-node-fg-color: #adb5bd;--mermaid-fg-color: #adb5bd;--mermaid-fg-color--lighter: #b5bcc4;--mermaid-fg-color--lightest: #bdc4ca;--mermaid-font-family: system-ui, -apple-system, Segoe UI, Roboto, Helvetica Neue, Arial, Noto Sans, Liberation Sans, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji;--mermaid-label-bg-color: #272b30;--mermaid-label-fg-color: #3e444a;--mermaid-node-bg-color: rgba(62, 68, 74, 0.1);--mermaid-node-fg-color: #adb5bd}@media print{:root{font-size:11pt}#quarto-sidebar,#TOC,.nav-page{display:none}.page-columns .content{grid-column-start:page-start}.fixed-top{position:relative}.panel-caption,.figure-caption,figcaption{color:#666}}.code-copy-button{position:absolute;top:0;right:0;border:0;margin-top:5px;margin-right:5px;background-color:rgba(0,0,0,0);z-index:3}.code-copy-button:focus{outline:none}.code-copy-button-tooltip{font-size:.75em}.code-copy-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}.code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}.code-copy-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}.code-copy-button-checked:hover>.bi::before{background-image:url('data:image/svg+xml,')}main ol ol,main ul ul,main ol ul,main ul ol{margin-bottom:1em}ul>li:not(:has(>p))>ul,ol>li:not(:has(>p))>ul,ul>li:not(:has(>p))>ol,ol>li:not(:has(>p))>ol{margin-bottom:0}ul>li:not(:has(>p))>ul>li:has(>p),ol>li:not(:has(>p))>ul>li:has(>p),ul>li:not(:has(>p))>ol>li:has(>p),ol>li:not(:has(>p))>ol>li:has(>p){margin-top:1rem}body{margin:0}main.page-columns>header>h1.title,main.page-columns>header>.title.h1{margin-bottom:0}@media(min-width: 992px){body .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.fullcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] 35px [page-end-inset page-end] 5fr [screen-end-inset] 1.5em}body.slimcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset] 35px [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.listing:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 850px - 3em )) [body-content-end] 3em [body-end] 50px [body-end-outset] minmax(0px, 250px) [page-end-inset] minmax(50px, 100px) [page-end] 1fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 175px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 175px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] minmax(25px, 50px) [page-start-inset] minmax(50px, 150px) [body-start-outset] minmax(25px, 50px) [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] minmax(25px, 50px) [body-end-outset] minmax(50px, 150px) [page-end-inset] minmax(25px, 50px) [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(50px, 100px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 50px [page-start-inset] minmax(50px, 150px) [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(450px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start] minmax(50px, 100px) [page-start-inset] 50px [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(0px, 200px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 50px [page-start-inset] minmax(50px, 150px) [body-start-outset] 50px [body-start] 1.5em [body-content-start] minmax(450px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(50px, 150px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] minmax(25px, 50px) [page-start-inset] minmax(50px, 150px) [body-start-outset] minmax(25px, 50px) [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] minmax(25px, 50px) [body-end-outset] minmax(50px, 150px) [page-end-inset] minmax(25px, 50px) [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}}@media(max-width: 991.98px){body .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.fullcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.slimcontent:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.listing:not(.floating):not(.docked) .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset] 5fr [body-start] 1.5em [body-content-start] minmax(500px, calc( 1250px - 3em )) [body-content-end body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 145px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start] 35px [page-start-inset] minmax(0px, 145px) [body-start-outset] 35px [body-start] 1.5em [body-content-start] minmax(450px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1.5em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(75px, 150px) [page-end-inset] 25px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 1000px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 800px - 3em )) [body-content-end] 1.5em [body-end body-end-outset page-end-inset page-end] 4fr [screen-end-inset] 1.5em [screen-end]}body.docked.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.docked.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(25px, 50px) [page-end-inset] 50px [page-end] 5fr [screen-end-inset] 1.5em [screen-end]}body.floating.slimcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 35px [body-end-outset] minmax(75px, 145px) [page-end-inset] 35px [page-end] 4fr [screen-end-inset] 1.5em [screen-end]}body.floating.listing .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset] 5fr [page-start page-start-inset body-start-outset body-start] 1em [body-content-start] minmax(500px, calc( 750px - 3em )) [body-content-end] 1.5em [body-end] 50px [body-end-outset] minmax(75px, 150px) [page-end-inset] 25px [page-end] 4fr [screen-end-inset] 1.5em [screen-end]}}@media(max-width: 767.98px){body .page-columns,body.fullcontent:not(.floating):not(.docked) .page-columns,body.slimcontent:not(.floating):not(.docked) .page-columns,body.docked .page-columns,body.docked.slimcontent .page-columns,body.docked.fullcontent .page-columns,body.floating .page-columns,body.floating.slimcontent .page-columns,body.floating.fullcontent .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}body:not(.floating):not(.docked) .page-columns.toc-left .page-columns{display:grid;gap:0;grid-template-columns:[screen-start] 1.5em [screen-start-inset page-start page-start-inset body-start-outset body-start body-content-start] minmax(0px, 1fr) [body-content-end body-end body-end-outset page-end-inset page-end screen-end-inset] 1.5em [screen-end]}nav[role=doc-toc]{display:none}}body,.page-row-navigation{grid-template-rows:[page-top] max-content [contents-top] max-content [contents-bottom] max-content [page-bottom]}.page-rows-contents{grid-template-rows:[content-top] minmax(max-content, 1fr) [content-bottom] minmax(60px, max-content) [page-bottom]}.page-full{grid-column:screen-start/screen-end !important}.page-columns>*{grid-column:body-content-start/body-content-end}.page-columns.column-page>*{grid-column:page-start/page-end}.page-columns.column-page-left>*{grid-column:page-start/body-content-end}.page-columns.column-page-right>*{grid-column:body-content-start/page-end}.page-rows{grid-auto-rows:auto}.header{grid-column:screen-start/screen-end;grid-row:page-top/contents-top}#quarto-content{padding:0;grid-column:screen-start/screen-end;grid-row:contents-top/contents-bottom}body.floating .sidebar.sidebar-navigation{grid-column:page-start/body-start;grid-row:content-top/page-bottom}body.docked .sidebar.sidebar-navigation{grid-column:screen-start/body-start;grid-row:content-top/page-bottom}.sidebar.toc-left{grid-column:page-start/body-start;grid-row:content-top/page-bottom}.sidebar.margin-sidebar{grid-column:body-end/page-end;grid-row:content-top/page-bottom}.page-columns .content{grid-column:body-content-start/body-content-end;grid-row:content-top/content-bottom;align-content:flex-start}.page-columns .page-navigation{grid-column:body-content-start/body-content-end;grid-row:content-bottom/page-bottom}.page-columns .footer{grid-column:screen-start/screen-end;grid-row:contents-bottom/page-bottom}.page-columns .column-body{grid-column:body-content-start/body-content-end}.page-columns .column-body-fullbleed{grid-column:body-start/body-end}.page-columns .column-body-outset{grid-column:body-start-outset/body-end-outset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset table{background:#272b30}.page-columns .column-body-outset-left{grid-column:body-start-outset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset-left table{background:#272b30}.page-columns .column-body-outset-right{grid-column:body-content-start/body-end-outset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-body-outset-right table{background:#272b30}.page-columns .column-page{grid-column:page-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page table{background:#272b30}.page-columns .column-page-inset{grid-column:page-start-inset/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset table{background:#272b30}.page-columns .column-page-inset-left{grid-column:page-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset-left table{background:#272b30}.page-columns .column-page-inset-right{grid-column:body-content-start/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-inset-right figcaption table{background:#272b30}.page-columns .column-page-left{grid-column:page-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-left table{background:#272b30}.page-columns .column-page-right{grid-column:body-content-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-page-right figcaption table{background:#272b30}#quarto-content.page-columns #quarto-margin-sidebar,#quarto-content.page-columns #quarto-sidebar{z-index:1}@media(max-width: 991.98px){#quarto-content.page-columns #quarto-margin-sidebar.collapse,#quarto-content.page-columns #quarto-sidebar.collapse,#quarto-content.page-columns #quarto-margin-sidebar.collapsing,#quarto-content.page-columns #quarto-sidebar.collapsing{z-index:1055}}#quarto-content.page-columns main.column-page,#quarto-content.page-columns main.column-page-right,#quarto-content.page-columns main.column-page-left{z-index:0}.page-columns .column-screen-inset{grid-column:screen-start-inset/screen-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset table{background:#272b30}.page-columns .column-screen-inset-left{grid-column:screen-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-left table{background:#272b30}.page-columns .column-screen-inset-right{grid-column:body-content-start/screen-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-right table{background:#272b30}.page-columns .column-screen{grid-column:screen-start/screen-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen table{background:#272b30}.page-columns .column-screen-left{grid-column:screen-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-left table{background:#272b30}.page-columns .column-screen-right{grid-column:body-content-start/screen-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-right table{background:#272b30}.page-columns .column-screen-inset-shaded{grid-column:screen-start/screen-end;padding:1em;background:#d6dade;z-index:998;transform:translate3d(0, 0, 0);margin-bottom:1em}.zindex-content{z-index:998;transform:translate3d(0, 0, 0)}.zindex-modal{z-index:1055;transform:translate3d(0, 0, 0)}.zindex-over-content{z-index:999;transform:translate3d(0, 0, 0)}img.img-fluid.column-screen,img.img-fluid.column-screen-inset-shaded,img.img-fluid.column-screen-inset,img.img-fluid.column-screen-inset-left,img.img-fluid.column-screen-inset-right,img.img-fluid.column-screen-left,img.img-fluid.column-screen-right{width:100%}@media(min-width: 992px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-end/page-end !important;z-index:998}.column-sidebar{grid-column:page-start/body-start !important;z-index:998}.column-leftmargin{grid-column:screen-start-inset/body-start !important;z-index:998}.no-row-height{height:1em;overflow:visible}}@media(max-width: 991.98px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-end/page-end !important;z-index:998}.no-row-height{height:1em;overflow:visible}.page-columns.page-full{overflow:visible}.page-columns.toc-left .margin-caption,.page-columns.toc-left div.aside,.page-columns.toc-left aside,.page-columns.toc-left .column-margin{grid-column:body-content-start/body-content-end !important;z-index:998;transform:translate3d(0, 0, 0)}.page-columns.toc-left .no-row-height{height:initial;overflow:initial}}@media(max-width: 767.98px){.margin-caption,div.aside,aside,.column-margin{grid-column:body-content-start/body-content-end !important;z-index:998;transform:translate3d(0, 0, 0)}.no-row-height{height:initial;overflow:initial}#quarto-margin-sidebar{display:none}#quarto-sidebar-toc-left{display:none}.hidden-sm{display:none}}.panel-grid{display:grid;grid-template-rows:repeat(1, 1fr);grid-template-columns:repeat(24, 1fr);gap:1em}.panel-grid .g-col-1{grid-column:auto/span 1}.panel-grid .g-col-2{grid-column:auto/span 2}.panel-grid .g-col-3{grid-column:auto/span 3}.panel-grid .g-col-4{grid-column:auto/span 4}.panel-grid .g-col-5{grid-column:auto/span 5}.panel-grid .g-col-6{grid-column:auto/span 6}.panel-grid .g-col-7{grid-column:auto/span 7}.panel-grid .g-col-8{grid-column:auto/span 8}.panel-grid .g-col-9{grid-column:auto/span 9}.panel-grid .g-col-10{grid-column:auto/span 10}.panel-grid .g-col-11{grid-column:auto/span 11}.panel-grid .g-col-12{grid-column:auto/span 12}.panel-grid .g-col-13{grid-column:auto/span 13}.panel-grid .g-col-14{grid-column:auto/span 14}.panel-grid .g-col-15{grid-column:auto/span 15}.panel-grid .g-col-16{grid-column:auto/span 16}.panel-grid .g-col-17{grid-column:auto/span 17}.panel-grid .g-col-18{grid-column:auto/span 18}.panel-grid .g-col-19{grid-column:auto/span 19}.panel-grid .g-col-20{grid-column:auto/span 20}.panel-grid .g-col-21{grid-column:auto/span 21}.panel-grid .g-col-22{grid-column:auto/span 22}.panel-grid .g-col-23{grid-column:auto/span 23}.panel-grid .g-col-24{grid-column:auto/span 24}.panel-grid .g-start-1{grid-column-start:1}.panel-grid .g-start-2{grid-column-start:2}.panel-grid .g-start-3{grid-column-start:3}.panel-grid .g-start-4{grid-column-start:4}.panel-grid .g-start-5{grid-column-start:5}.panel-grid .g-start-6{grid-column-start:6}.panel-grid .g-start-7{grid-column-start:7}.panel-grid .g-start-8{grid-column-start:8}.panel-grid .g-start-9{grid-column-start:9}.panel-grid .g-start-10{grid-column-start:10}.panel-grid .g-start-11{grid-column-start:11}.panel-grid .g-start-12{grid-column-start:12}.panel-grid .g-start-13{grid-column-start:13}.panel-grid .g-start-14{grid-column-start:14}.panel-grid .g-start-15{grid-column-start:15}.panel-grid .g-start-16{grid-column-start:16}.panel-grid .g-start-17{grid-column-start:17}.panel-grid .g-start-18{grid-column-start:18}.panel-grid .g-start-19{grid-column-start:19}.panel-grid .g-start-20{grid-column-start:20}.panel-grid .g-start-21{grid-column-start:21}.panel-grid .g-start-22{grid-column-start:22}.panel-grid .g-start-23{grid-column-start:23}@media(min-width: 576px){.panel-grid .g-col-sm-1{grid-column:auto/span 1}.panel-grid .g-col-sm-2{grid-column:auto/span 2}.panel-grid .g-col-sm-3{grid-column:auto/span 3}.panel-grid .g-col-sm-4{grid-column:auto/span 4}.panel-grid .g-col-sm-5{grid-column:auto/span 5}.panel-grid .g-col-sm-6{grid-column:auto/span 6}.panel-grid .g-col-sm-7{grid-column:auto/span 7}.panel-grid .g-col-sm-8{grid-column:auto/span 8}.panel-grid .g-col-sm-9{grid-column:auto/span 9}.panel-grid .g-col-sm-10{grid-column:auto/span 10}.panel-grid .g-col-sm-11{grid-column:auto/span 11}.panel-grid .g-col-sm-12{grid-column:auto/span 12}.panel-grid .g-col-sm-13{grid-column:auto/span 13}.panel-grid .g-col-sm-14{grid-column:auto/span 14}.panel-grid .g-col-sm-15{grid-column:auto/span 15}.panel-grid .g-col-sm-16{grid-column:auto/span 16}.panel-grid .g-col-sm-17{grid-column:auto/span 17}.panel-grid .g-col-sm-18{grid-column:auto/span 18}.panel-grid .g-col-sm-19{grid-column:auto/span 19}.panel-grid .g-col-sm-20{grid-column:auto/span 20}.panel-grid .g-col-sm-21{grid-column:auto/span 21}.panel-grid .g-col-sm-22{grid-column:auto/span 22}.panel-grid .g-col-sm-23{grid-column:auto/span 23}.panel-grid .g-col-sm-24{grid-column:auto/span 24}.panel-grid .g-start-sm-1{grid-column-start:1}.panel-grid .g-start-sm-2{grid-column-start:2}.panel-grid .g-start-sm-3{grid-column-start:3}.panel-grid .g-start-sm-4{grid-column-start:4}.panel-grid .g-start-sm-5{grid-column-start:5}.panel-grid .g-start-sm-6{grid-column-start:6}.panel-grid .g-start-sm-7{grid-column-start:7}.panel-grid .g-start-sm-8{grid-column-start:8}.panel-grid .g-start-sm-9{grid-column-start:9}.panel-grid .g-start-sm-10{grid-column-start:10}.panel-grid .g-start-sm-11{grid-column-start:11}.panel-grid .g-start-sm-12{grid-column-start:12}.panel-grid .g-start-sm-13{grid-column-start:13}.panel-grid .g-start-sm-14{grid-column-start:14}.panel-grid .g-start-sm-15{grid-column-start:15}.panel-grid .g-start-sm-16{grid-column-start:16}.panel-grid .g-start-sm-17{grid-column-start:17}.panel-grid .g-start-sm-18{grid-column-start:18}.panel-grid .g-start-sm-19{grid-column-start:19}.panel-grid .g-start-sm-20{grid-column-start:20}.panel-grid .g-start-sm-21{grid-column-start:21}.panel-grid .g-start-sm-22{grid-column-start:22}.panel-grid .g-start-sm-23{grid-column-start:23}}@media(min-width: 768px){.panel-grid .g-col-md-1{grid-column:auto/span 1}.panel-grid .g-col-md-2{grid-column:auto/span 2}.panel-grid .g-col-md-3{grid-column:auto/span 3}.panel-grid .g-col-md-4{grid-column:auto/span 4}.panel-grid .g-col-md-5{grid-column:auto/span 5}.panel-grid .g-col-md-6{grid-column:auto/span 6}.panel-grid .g-col-md-7{grid-column:auto/span 7}.panel-grid .g-col-md-8{grid-column:auto/span 8}.panel-grid .g-col-md-9{grid-column:auto/span 9}.panel-grid .g-col-md-10{grid-column:auto/span 10}.panel-grid .g-col-md-11{grid-column:auto/span 11}.panel-grid .g-col-md-12{grid-column:auto/span 12}.panel-grid .g-col-md-13{grid-column:auto/span 13}.panel-grid .g-col-md-14{grid-column:auto/span 14}.panel-grid .g-col-md-15{grid-column:auto/span 15}.panel-grid .g-col-md-16{grid-column:auto/span 16}.panel-grid .g-col-md-17{grid-column:auto/span 17}.panel-grid .g-col-md-18{grid-column:auto/span 18}.panel-grid .g-col-md-19{grid-column:auto/span 19}.panel-grid .g-col-md-20{grid-column:auto/span 20}.panel-grid .g-col-md-21{grid-column:auto/span 21}.panel-grid .g-col-md-22{grid-column:auto/span 22}.panel-grid .g-col-md-23{grid-column:auto/span 23}.panel-grid .g-col-md-24{grid-column:auto/span 24}.panel-grid .g-start-md-1{grid-column-start:1}.panel-grid .g-start-md-2{grid-column-start:2}.panel-grid .g-start-md-3{grid-column-start:3}.panel-grid .g-start-md-4{grid-column-start:4}.panel-grid .g-start-md-5{grid-column-start:5}.panel-grid .g-start-md-6{grid-column-start:6}.panel-grid .g-start-md-7{grid-column-start:7}.panel-grid .g-start-md-8{grid-column-start:8}.panel-grid .g-start-md-9{grid-column-start:9}.panel-grid .g-start-md-10{grid-column-start:10}.panel-grid .g-start-md-11{grid-column-start:11}.panel-grid .g-start-md-12{grid-column-start:12}.panel-grid .g-start-md-13{grid-column-start:13}.panel-grid .g-start-md-14{grid-column-start:14}.panel-grid .g-start-md-15{grid-column-start:15}.panel-grid .g-start-md-16{grid-column-start:16}.panel-grid .g-start-md-17{grid-column-start:17}.panel-grid .g-start-md-18{grid-column-start:18}.panel-grid .g-start-md-19{grid-column-start:19}.panel-grid .g-start-md-20{grid-column-start:20}.panel-grid .g-start-md-21{grid-column-start:21}.panel-grid .g-start-md-22{grid-column-start:22}.panel-grid .g-start-md-23{grid-column-start:23}}@media(min-width: 992px){.panel-grid .g-col-lg-1{grid-column:auto/span 1}.panel-grid .g-col-lg-2{grid-column:auto/span 2}.panel-grid .g-col-lg-3{grid-column:auto/span 3}.panel-grid .g-col-lg-4{grid-column:auto/span 4}.panel-grid .g-col-lg-5{grid-column:auto/span 5}.panel-grid .g-col-lg-6{grid-column:auto/span 6}.panel-grid .g-col-lg-7{grid-column:auto/span 7}.panel-grid .g-col-lg-8{grid-column:auto/span 8}.panel-grid .g-col-lg-9{grid-column:auto/span 9}.panel-grid .g-col-lg-10{grid-column:auto/span 10}.panel-grid .g-col-lg-11{grid-column:auto/span 11}.panel-grid .g-col-lg-12{grid-column:auto/span 12}.panel-grid .g-col-lg-13{grid-column:auto/span 13}.panel-grid .g-col-lg-14{grid-column:auto/span 14}.panel-grid .g-col-lg-15{grid-column:auto/span 15}.panel-grid .g-col-lg-16{grid-column:auto/span 16}.panel-grid .g-col-lg-17{grid-column:auto/span 17}.panel-grid .g-col-lg-18{grid-column:auto/span 18}.panel-grid .g-col-lg-19{grid-column:auto/span 19}.panel-grid .g-col-lg-20{grid-column:auto/span 20}.panel-grid .g-col-lg-21{grid-column:auto/span 21}.panel-grid .g-col-lg-22{grid-column:auto/span 22}.panel-grid .g-col-lg-23{grid-column:auto/span 23}.panel-grid .g-col-lg-24{grid-column:auto/span 24}.panel-grid .g-start-lg-1{grid-column-start:1}.panel-grid .g-start-lg-2{grid-column-start:2}.panel-grid .g-start-lg-3{grid-column-start:3}.panel-grid .g-start-lg-4{grid-column-start:4}.panel-grid .g-start-lg-5{grid-column-start:5}.panel-grid .g-start-lg-6{grid-column-start:6}.panel-grid .g-start-lg-7{grid-column-start:7}.panel-grid .g-start-lg-8{grid-column-start:8}.panel-grid .g-start-lg-9{grid-column-start:9}.panel-grid .g-start-lg-10{grid-column-start:10}.panel-grid .g-start-lg-11{grid-column-start:11}.panel-grid .g-start-lg-12{grid-column-start:12}.panel-grid .g-start-lg-13{grid-column-start:13}.panel-grid .g-start-lg-14{grid-column-start:14}.panel-grid .g-start-lg-15{grid-column-start:15}.panel-grid .g-start-lg-16{grid-column-start:16}.panel-grid .g-start-lg-17{grid-column-start:17}.panel-grid .g-start-lg-18{grid-column-start:18}.panel-grid .g-start-lg-19{grid-column-start:19}.panel-grid .g-start-lg-20{grid-column-start:20}.panel-grid .g-start-lg-21{grid-column-start:21}.panel-grid .g-start-lg-22{grid-column-start:22}.panel-grid .g-start-lg-23{grid-column-start:23}}@media(min-width: 1200px){.panel-grid .g-col-xl-1{grid-column:auto/span 1}.panel-grid .g-col-xl-2{grid-column:auto/span 2}.panel-grid .g-col-xl-3{grid-column:auto/span 3}.panel-grid .g-col-xl-4{grid-column:auto/span 4}.panel-grid .g-col-xl-5{grid-column:auto/span 5}.panel-grid .g-col-xl-6{grid-column:auto/span 6}.panel-grid .g-col-xl-7{grid-column:auto/span 7}.panel-grid .g-col-xl-8{grid-column:auto/span 8}.panel-grid .g-col-xl-9{grid-column:auto/span 9}.panel-grid .g-col-xl-10{grid-column:auto/span 10}.panel-grid .g-col-xl-11{grid-column:auto/span 11}.panel-grid .g-col-xl-12{grid-column:auto/span 12}.panel-grid .g-col-xl-13{grid-column:auto/span 13}.panel-grid .g-col-xl-14{grid-column:auto/span 14}.panel-grid .g-col-xl-15{grid-column:auto/span 15}.panel-grid .g-col-xl-16{grid-column:auto/span 16}.panel-grid .g-col-xl-17{grid-column:auto/span 17}.panel-grid .g-col-xl-18{grid-column:auto/span 18}.panel-grid .g-col-xl-19{grid-column:auto/span 19}.panel-grid .g-col-xl-20{grid-column:auto/span 20}.panel-grid .g-col-xl-21{grid-column:auto/span 21}.panel-grid .g-col-xl-22{grid-column:auto/span 22}.panel-grid .g-col-xl-23{grid-column:auto/span 23}.panel-grid .g-col-xl-24{grid-column:auto/span 24}.panel-grid .g-start-xl-1{grid-column-start:1}.panel-grid .g-start-xl-2{grid-column-start:2}.panel-grid .g-start-xl-3{grid-column-start:3}.panel-grid .g-start-xl-4{grid-column-start:4}.panel-grid .g-start-xl-5{grid-column-start:5}.panel-grid .g-start-xl-6{grid-column-start:6}.panel-grid .g-start-xl-7{grid-column-start:7}.panel-grid .g-start-xl-8{grid-column-start:8}.panel-grid .g-start-xl-9{grid-column-start:9}.panel-grid .g-start-xl-10{grid-column-start:10}.panel-grid .g-start-xl-11{grid-column-start:11}.panel-grid .g-start-xl-12{grid-column-start:12}.panel-grid .g-start-xl-13{grid-column-start:13}.panel-grid .g-start-xl-14{grid-column-start:14}.panel-grid .g-start-xl-15{grid-column-start:15}.panel-grid .g-start-xl-16{grid-column-start:16}.panel-grid .g-start-xl-17{grid-column-start:17}.panel-grid .g-start-xl-18{grid-column-start:18}.panel-grid .g-start-xl-19{grid-column-start:19}.panel-grid .g-start-xl-20{grid-column-start:20}.panel-grid .g-start-xl-21{grid-column-start:21}.panel-grid .g-start-xl-22{grid-column-start:22}.panel-grid .g-start-xl-23{grid-column-start:23}}@media(min-width: 1400px){.panel-grid .g-col-xxl-1{grid-column:auto/span 1}.panel-grid .g-col-xxl-2{grid-column:auto/span 2}.panel-grid .g-col-xxl-3{grid-column:auto/span 3}.panel-grid .g-col-xxl-4{grid-column:auto/span 4}.panel-grid .g-col-xxl-5{grid-column:auto/span 5}.panel-grid .g-col-xxl-6{grid-column:auto/span 6}.panel-grid .g-col-xxl-7{grid-column:auto/span 7}.panel-grid .g-col-xxl-8{grid-column:auto/span 8}.panel-grid .g-col-xxl-9{grid-column:auto/span 9}.panel-grid .g-col-xxl-10{grid-column:auto/span 10}.panel-grid .g-col-xxl-11{grid-column:auto/span 11}.panel-grid .g-col-xxl-12{grid-column:auto/span 12}.panel-grid .g-col-xxl-13{grid-column:auto/span 13}.panel-grid .g-col-xxl-14{grid-column:auto/span 14}.panel-grid .g-col-xxl-15{grid-column:auto/span 15}.panel-grid .g-col-xxl-16{grid-column:auto/span 16}.panel-grid .g-col-xxl-17{grid-column:auto/span 17}.panel-grid .g-col-xxl-18{grid-column:auto/span 18}.panel-grid .g-col-xxl-19{grid-column:auto/span 19}.panel-grid .g-col-xxl-20{grid-column:auto/span 20}.panel-grid .g-col-xxl-21{grid-column:auto/span 21}.panel-grid .g-col-xxl-22{grid-column:auto/span 22}.panel-grid .g-col-xxl-23{grid-column:auto/span 23}.panel-grid .g-col-xxl-24{grid-column:auto/span 24}.panel-grid .g-start-xxl-1{grid-column-start:1}.panel-grid .g-start-xxl-2{grid-column-start:2}.panel-grid .g-start-xxl-3{grid-column-start:3}.panel-grid .g-start-xxl-4{grid-column-start:4}.panel-grid .g-start-xxl-5{grid-column-start:5}.panel-grid .g-start-xxl-6{grid-column-start:6}.panel-grid .g-start-xxl-7{grid-column-start:7}.panel-grid .g-start-xxl-8{grid-column-start:8}.panel-grid .g-start-xxl-9{grid-column-start:9}.panel-grid .g-start-xxl-10{grid-column-start:10}.panel-grid .g-start-xxl-11{grid-column-start:11}.panel-grid .g-start-xxl-12{grid-column-start:12}.panel-grid .g-start-xxl-13{grid-column-start:13}.panel-grid .g-start-xxl-14{grid-column-start:14}.panel-grid .g-start-xxl-15{grid-column-start:15}.panel-grid .g-start-xxl-16{grid-column-start:16}.panel-grid .g-start-xxl-17{grid-column-start:17}.panel-grid .g-start-xxl-18{grid-column-start:18}.panel-grid .g-start-xxl-19{grid-column-start:19}.panel-grid .g-start-xxl-20{grid-column-start:20}.panel-grid .g-start-xxl-21{grid-column-start:21}.panel-grid .g-start-xxl-22{grid-column-start:22}.panel-grid .g-start-xxl-23{grid-column-start:23}}main{margin-top:1em;margin-bottom:1em}h1,.h1,h2,.h2{opacity:.9;margin-top:2rem;margin-bottom:1rem;font-weight:600}h1.title,.title.h1{margin-top:0}h2,.h2{border-bottom:1px solid rgba(0,0,0,.6);padding-bottom:.5rem}h3,.h3{font-weight:600}h3,.h3,h4,.h4{opacity:.9;margin-top:1.5rem}h5,.h5,h6,.h6{opacity:.9}.header-section-number{color:#82888e}.nav-link.active .header-section-number{color:inherit}mark,.mark{padding:0em}.panel-caption,caption,.figure-caption{font-size:.9rem}.panel-caption,.figure-caption,figcaption{color:#82888e}.table-caption,caption{color:#adb5bd}.quarto-layout-cell[data-ref-parent] caption{color:#82888e}.column-margin figcaption,.margin-caption,div.aside,aside,.column-margin{color:#82888e;font-size:.825rem}.panel-caption.margin-caption{text-align:inherit}.column-margin.column-container p{margin-bottom:0}.column-margin.column-container>*:not(.collapse){padding-top:.5em;padding-bottom:.5em;display:block}.column-margin.column-container>*.collapse:not(.show){display:none}@media(min-width: 768px){.column-margin.column-container .callout-margin-content:first-child{margin-top:4.5em}.column-margin.column-container .callout-margin-content-simple:first-child{margin-top:3.5em}}.margin-caption>*{padding-top:.5em;padding-bottom:.5em}@media(max-width: 767.98px){.quarto-layout-row{flex-direction:column}}.nav-tabs .nav-item{margin-top:1px;cursor:pointer}.tab-content{margin-top:0px;border-left:rgba(0,0,0,.6) 1px solid;border-right:rgba(0,0,0,.6) 1px solid;border-bottom:rgba(0,0,0,.6) 1px solid;margin-left:0;padding:1em;margin-bottom:1em}@media(max-width: 767.98px){.layout-sidebar{margin-left:0;margin-right:0}}.panel-sidebar,.panel-sidebar .form-control,.panel-input,.panel-input .form-control,.selectize-dropdown{font-size:.9rem}.panel-sidebar .form-control,.panel-input .form-control{padding-top:.1rem}.tab-pane div.sourceCode{margin-top:0px}.tab-pane>p{padding-top:1em}.tab-content>.tab-pane:not(.active){display:none !important}div.sourceCode{background-color:rgba(37,41,46,.65);border:1px solid rgba(37,41,46,.65);border-radius:.25rem}pre.sourceCode{background-color:rgba(0,0,0,0)}pre.sourceCode{border:none;font-size:.875em;overflow:visible !important;padding:.4em}.callout pre.sourceCode{padding-left:0}div.sourceCode{overflow-y:hidden}.callout div.sourceCode{margin-left:initial}.blockquote{font-size:inherit;padding-left:1rem;padding-right:1.5rem;color:#82888e}.blockquote h1:first-child,.blockquote .h1:first-child,.blockquote h2:first-child,.blockquote .h2:first-child,.blockquote h3:first-child,.blockquote .h3:first-child,.blockquote h4:first-child,.blockquote .h4:first-child,.blockquote h5:first-child,.blockquote .h5:first-child{margin-top:0}pre{background-color:initial;padding:initial;border:initial}p code:not(.sourceCode),li code:not(.sourceCode),td code:not(.sourceCode){background-color:#2c3136;padding:.2em}nav p code:not(.sourceCode),nav li code:not(.sourceCode),nav td code:not(.sourceCode){background-color:rgba(0,0,0,0);padding:0}td code:not(.sourceCode){white-space:pre-wrap}#quarto-embedded-source-code-modal>.modal-dialog{max-width:1000px;padding-left:1.75rem;padding-right:1.75rem}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-body{padding:0}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-body div.sourceCode{margin:0;padding:.2rem .2rem;border-radius:0px;border:none}#quarto-embedded-source-code-modal>.modal-dialog>.modal-content>.modal-header{padding:.7rem}.code-tools-button{font-size:1rem;padding:.15rem .15rem;margin-left:5px;color:#6c757d;background-color:rgba(0,0,0,0);transition:initial;cursor:pointer}.code-tools-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}.code-tools-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}#quarto-embedded-source-code-modal .code-copy-button>.bi::before{background-image:url('data:image/svg+xml,')}#quarto-embedded-source-code-modal .code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}.sidebar{will-change:top;transition:top 200ms linear;position:sticky;overflow-y:auto;padding-top:1.2em;max-height:100vh}.sidebar.toc-left,.sidebar.margin-sidebar{top:0px;padding-top:1em}.sidebar.toc-left>*,.sidebar.margin-sidebar>*{padding-top:.5em}.sidebar.quarto-banner-title-block-sidebar>*{padding-top:1.65em}figure .quarto-notebook-link{margin-top:.5em}.quarto-notebook-link{font-size:.75em;color:#6c757d;margin-bottom:1em;text-decoration:none;display:block}.quarto-notebook-link:hover{text-decoration:underline;color:#fff}.quarto-notebook-link::before{display:inline-block;height:.75rem;width:.75rem;margin-bottom:0em;margin-right:.25em;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:.75rem .75rem}.quarto-alternate-notebooks i.bi,.quarto-alternate-formats i.bi{margin-right:.4em}.quarto-notebook .cell-container{display:flex}.quarto-notebook .cell-container .cell{flex-grow:4}.quarto-notebook .cell-container .cell-decorator{padding-top:1.5em;padding-right:1em;text-align:right}.quarto-notebook h2,.quarto-notebook .h2{border-bottom:none}.sidebar .quarto-alternate-formats a,.sidebar .quarto-alternate-notebooks a{text-decoration:none}.sidebar .quarto-alternate-formats a:hover,.sidebar .quarto-alternate-notebooks a:hover{color:#fff}.sidebar .quarto-alternate-notebooks h2,.sidebar .quarto-alternate-notebooks .h2,.sidebar .quarto-alternate-formats h2,.sidebar .quarto-alternate-formats .h2,.sidebar nav[role=doc-toc]>h2,.sidebar nav[role=doc-toc]>.h2{font-size:.875rem;font-weight:400;margin-bottom:.5rem;margin-top:.3rem;font-family:inherit;border-bottom:0;padding-bottom:0;padding-top:0px}.sidebar .quarto-alternate-notebooks h2,.sidebar .quarto-alternate-notebooks .h2,.sidebar .quarto-alternate-formats h2,.sidebar .quarto-alternate-formats .h2{margin-top:1rem}.sidebar nav[role=doc-toc]>ul a{border-left:1px solid #e9ecef;padding-left:.6rem}.sidebar .quarto-alternate-notebooks h2>ul a,.sidebar .quarto-alternate-notebooks .h2>ul a,.sidebar .quarto-alternate-formats h2>ul a,.sidebar .quarto-alternate-formats .h2>ul a{border-left:none;padding-left:.6rem}.sidebar .quarto-alternate-notebooks ul a:empty,.sidebar .quarto-alternate-formats ul a:empty,.sidebar nav[role=doc-toc]>ul a:empty{display:none}.sidebar .quarto-alternate-notebooks ul,.sidebar .quarto-alternate-formats ul,.sidebar nav[role=doc-toc] ul{padding-left:0;list-style:none;font-size:.875rem;font-weight:300}.sidebar .quarto-alternate-notebooks ul li a,.sidebar .quarto-alternate-formats ul li a,.sidebar nav[role=doc-toc]>ul li a{line-height:1.1rem;padding-bottom:.2rem;padding-top:.2rem;color:inherit}.sidebar nav[role=doc-toc] ul>li>ul>li>a{padding-left:1.2em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>a{padding-left:2.4em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>a{padding-left:3.6em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>ul>li>a{padding-left:4.8em}.sidebar nav[role=doc-toc] ul>li>ul>li>ul>li>ul>li>ul>li>ul>li>a{padding-left:6em}.sidebar nav[role=doc-toc] ul>li>a.active,.sidebar nav[role=doc-toc] ul>li>ul>li>a.active{border-left:1px solid #fff;color:#fff !important}.sidebar nav[role=doc-toc] ul>li>a:hover,.sidebar nav[role=doc-toc] ul>li>ul>li>a:hover{color:#fff !important}kbd,.kbd{color:#adb5bd;background-color:#f8f9fa;border:1px solid;border-radius:5px;border-color:rgba(0,0,0,.6)}div.hanging-indent{margin-left:1em;text-indent:-1em}.citation a,.footnote-ref{text-decoration:none}.footnotes ol{padding-left:1em}.tippy-content>*{margin-bottom:.7em}.tippy-content>*:last-child{margin-bottom:0}.table a{word-break:break-word}.table>thead{border-top-width:1px;border-top-color:rgba(0,0,0,.6);border-bottom:1px solid #d6dade}.callout{margin-top:1.25rem;margin-bottom:1.25rem;border-radius:.25rem;overflow-wrap:break-word}.callout .callout-title-container{overflow-wrap:anywhere}.callout.callout-style-simple{padding:.4em .7em;border-left:5px solid;border-right:1px solid rgba(0,0,0,.6);border-top:1px solid rgba(0,0,0,.6);border-bottom:1px solid rgba(0,0,0,.6)}.callout.callout-style-default{border-left:5px solid;border-right:1px solid rgba(0,0,0,.6);border-top:1px solid rgba(0,0,0,.6);border-bottom:1px solid rgba(0,0,0,.6)}.callout .callout-body-container{flex-grow:1}.callout.callout-style-simple .callout-body{font-size:.9rem;font-weight:400}.callout.callout-style-default .callout-body{font-size:.9rem;font-weight:400}.callout.callout-titled .callout-body{margin-top:.2em}.callout:not(.no-icon).callout-titled.callout-style-simple .callout-body{padding-left:1.6em}.callout.callout-titled>.callout-header{padding-top:.2em;margin-bottom:-0.2em}.callout.callout-style-simple>div.callout-header{border-bottom:none;font-size:.9rem;font-weight:600;opacity:75%}.callout.callout-style-default>div.callout-header{border-bottom:none;font-weight:600;opacity:85%;font-size:.9rem;padding-left:.5em;padding-right:.5em}.callout.callout-style-default div.callout-body{padding-left:.5em;padding-right:.5em}.callout.callout-style-default div.callout-body>:first-child{margin-top:.5em}.callout>div.callout-header[data-bs-toggle=collapse]{cursor:pointer}.callout.callout-style-default .callout-header[aria-expanded=false],.callout.callout-style-default .callout-header[aria-expanded=true]{padding-top:0px;margin-bottom:0px;align-items:center}.callout.callout-titled .callout-body>:last-child:not(.sourceCode),.callout.callout-titled .callout-body>div>:last-child:not(.sourceCode){margin-bottom:.5rem}.callout:not(.callout-titled) .callout-body>:first-child,.callout:not(.callout-titled) .callout-body>div>:first-child{margin-top:.25rem}.callout:not(.callout-titled) .callout-body>:last-child,.callout:not(.callout-titled) .callout-body>div>:last-child{margin-bottom:.2rem}.callout.callout-style-simple .callout-icon::before,.callout.callout-style-simple .callout-toggle::before{height:1rem;width:1rem;display:inline-block;content:"";background-repeat:no-repeat;background-size:1rem 1rem}.callout.callout-style-default .callout-icon::before,.callout.callout-style-default .callout-toggle::before{height:.9rem;width:.9rem;display:inline-block;content:"";background-repeat:no-repeat;background-size:.9rem .9rem}.callout.callout-style-default .callout-toggle::before{margin-top:5px}.callout .callout-btn-toggle .callout-toggle::before{transition:transform .2s linear}.callout .callout-header[aria-expanded=false] .callout-toggle::before{transform:rotate(-90deg)}.callout .callout-header[aria-expanded=true] .callout-toggle::before{transform:none}.callout.callout-style-simple:not(.no-icon) div.callout-icon-container{padding-top:.2em;padding-right:.55em}.callout.callout-style-default:not(.no-icon) div.callout-icon-container{padding-top:.1em;padding-right:.35em}.callout.callout-style-default:not(.no-icon) div.callout-title-container{margin-top:-1px}.callout.callout-style-default.callout-caution:not(.no-icon) div.callout-icon-container{padding-top:.3em;padding-right:.35em}.callout>.callout-body>.callout-icon-container>.no-icon,.callout>.callout-header>.callout-icon-container>.no-icon{display:none}div.callout.callout{border-left-color:#6c757d}div.callout.callout-style-default>.callout-header{background-color:#6c757d}div.callout-note.callout{border-left-color:#007bff}div.callout-note.callout-style-default>.callout-header{background-color:#00254d}div.callout-note:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-note.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-note .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-tip.callout{border-left-color:#62c462}div.callout-tip.callout-style-default>.callout-header{background-color:#1d3b1d}div.callout-tip:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-tip.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-tip .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-warning.callout{border-left-color:#f89406}div.callout-warning.callout-style-default>.callout-header{background-color:#4a2c02}div.callout-warning:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-warning.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-warning .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-caution.callout{border-left-color:#fd7e14}div.callout-caution.callout-style-default>.callout-header{background-color:#4c2606}div.callout-caution:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-caution.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-caution .callout-toggle::before{background-image:url('data:image/svg+xml,')}div.callout-important.callout{border-left-color:#ee5f5b}div.callout-important.callout-style-default>.callout-header{background-color:#471d1b}div.callout-important:not(.callout-titled) .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-important.callout-titled .callout-icon::before{background-image:url('data:image/svg+xml,');}div.callout-important .callout-toggle::before{background-image:url('data:image/svg+xml,')}.quarto-toggle-container{display:flex;align-items:center}.quarto-reader-toggle .bi::before,.quarto-color-scheme-toggle .bi::before{display:inline-block;height:1rem;width:1rem;content:"";background-repeat:no-repeat;background-size:1rem 1rem}.sidebar-navigation{padding-left:20px}.navbar .quarto-color-scheme-toggle:not(.alternate) .bi::before{background-image:url('data:image/svg+xml,')}.navbar .quarto-color-scheme-toggle.alternate .bi::before{background-image:url('data:image/svg+xml,')}.sidebar-navigation .quarto-color-scheme-toggle:not(.alternate) .bi::before{background-image:url('data:image/svg+xml,')}.sidebar-navigation .quarto-color-scheme-toggle.alternate .bi::before{background-image:url('data:image/svg+xml,')}.quarto-sidebar-toggle{border-color:#dee2e6;border-bottom-left-radius:.25rem;border-bottom-right-radius:.25rem;border-style:solid;border-width:1px;overflow:hidden;border-top-width:0px;padding-top:0px !important}.quarto-sidebar-toggle-title{cursor:pointer;padding-bottom:2px;margin-left:.25em;text-align:center;font-weight:400;font-size:.775em}#quarto-content .quarto-sidebar-toggle{background:#2b2f34}#quarto-content .quarto-sidebar-toggle-title{color:#adb5bd}.quarto-sidebar-toggle-icon{color:#dee2e6;margin-right:.5em;float:right;transition:transform .2s ease}.quarto-sidebar-toggle-icon::before{padding-top:5px}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-icon{transform:rotate(-180deg)}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-title{border-bottom:solid #dee2e6 1px}.quarto-sidebar-toggle-contents{background-color:#272b30;padding-right:10px;padding-left:10px;margin-top:0px !important;transition:max-height .5s ease}.quarto-sidebar-toggle.expanded .quarto-sidebar-toggle-contents{padding-top:1em;padding-bottom:10px}.quarto-sidebar-toggle:not(.expanded) .quarto-sidebar-toggle-contents{padding-top:0px !important;padding-bottom:0px}nav[role=doc-toc]{z-index:1020}#quarto-sidebar>*,nav[role=doc-toc]>*{transition:opacity .1s ease,border .1s ease}#quarto-sidebar.slow>*,nav[role=doc-toc].slow>*{transition:opacity .4s ease,border .4s ease}.quarto-color-scheme-toggle:not(.alternate).top-right .bi::before{background-image:url('data:image/svg+xml,')}.quarto-color-scheme-toggle.alternate.top-right .bi::before{background-image:url('data:image/svg+xml,')}#quarto-appendix.default{border-top:1px solid #dee2e6}#quarto-appendix.default{background-color:#272b30;padding-top:1.5em;margin-top:2em;z-index:998}#quarto-appendix.default .quarto-appendix-heading{margin-top:0;line-height:1.4em;font-weight:600;opacity:.9;border-bottom:none;margin-bottom:0}#quarto-appendix.default .footnotes ol,#quarto-appendix.default .footnotes ol li>p:last-of-type,#quarto-appendix.default .quarto-appendix-contents>p:last-of-type{margin-bottom:0}#quarto-appendix.default .quarto-appendix-secondary-label{margin-bottom:.4em}#quarto-appendix.default .quarto-appendix-bibtex{font-size:.7em;padding:1em;border:solid 1px #dee2e6;margin-bottom:1em}#quarto-appendix.default .quarto-appendix-bibtex code.sourceCode{white-space:pre-wrap}#quarto-appendix.default .quarto-appendix-citeas{font-size:.9em;padding:1em;border:solid 1px #dee2e6;margin-bottom:1em}#quarto-appendix.default .quarto-appendix-heading{font-size:1em !important}#quarto-appendix.default *[role=doc-endnotes]>ol,#quarto-appendix.default .quarto-appendix-contents>*:not(h2):not(.h2){font-size:.9em}#quarto-appendix.default section{padding-bottom:1.5em}#quarto-appendix.default section *[role=doc-endnotes],#quarto-appendix.default section>*:not(a){opacity:.9;word-wrap:break-word}.btn.btn-quarto,div.cell-output-display .btn-quarto{color:#fefefe;background-color:#6c757d;background-image:var(--bs-gradient);border-color:#6c757d;box-shadow:inset 0 1px 0 rgba(255,255,255,.15),0 1px 1px rgba(0,0,0,.075)}.btn.btn-quarto:hover,div.cell-output-display .btn-quarto:hover{color:#fefefe;background-color:#828a91;background-image:var(--bs-gradient);border-color:#7b838a}.btn-check:focus+.btn.btn-quarto,.btn.btn-quarto:focus,.btn-check:focus+div.cell-output-display .btn-quarto,div.cell-output-display .btn-quarto:focus{color:#fefefe;background-color:#828a91;background-image:var(--bs-gradient);border-color:#7b838a;box-shadow:inset 0 1px 0 rgba(255,255,255,.15),0 1px 1px rgba(0,0,0,.075),0 0 0 .25rem rgba(130,138,144,.5)}.btn-check:checked+.btn.btn-quarto,.btn-check:active+.btn.btn-quarto,.btn.btn-quarto:active,.btn.btn-quarto.active,.show>.btn.btn-quarto.dropdown-toggle,.btn-check:checked+div.cell-output-display .btn-quarto,.btn-check:active+div.cell-output-display .btn-quarto,div.cell-output-display .btn-quarto:active,div.cell-output-display .btn-quarto.active,.show>div.cell-output-display .btn-quarto.dropdown-toggle{color:#fff;background-color:#899197;background-image:none;border-color:#7b838a}.btn-check:checked+.btn.btn-quarto:focus,.btn-check:active+.btn.btn-quarto:focus,.btn.btn-quarto:active:focus,.btn.btn-quarto.active:focus,.show>.btn.btn-quarto.dropdown-toggle:focus,.btn-check:checked+div.cell-output-display .btn-quarto:focus,.btn-check:active+div.cell-output-display .btn-quarto:focus,div.cell-output-display .btn-quarto:active:focus,div.cell-output-display .btn-quarto.active:focus,.show>div.cell-output-display .btn-quarto.dropdown-toggle:focus{box-shadow:inset 0 3px 5px rgba(0,0,0,.125),0 0 0 .25rem rgba(130,138,144,.5)}.btn.btn-quarto:disabled,.btn.btn-quarto.disabled,div.cell-output-display .btn-quarto:disabled,div.cell-output-display .btn-quarto.disabled{color:#fff;background-color:#6c757d;background-image:none;border-color:#6c757d}nav.quarto-secondary-nav.color-navbar{background-color:#3e444a;color:#d8dadb}nav.quarto-secondary-nav.color-navbar h1,nav.quarto-secondary-nav.color-navbar .h1,nav.quarto-secondary-nav.color-navbar .quarto-btn-toggle{color:#d8dadb}@media(max-width: 991.98px){body.nav-sidebar .quarto-title-banner{margin-bottom:0;padding-bottom:0}body.nav-sidebar #title-block-header{margin-block-end:0}}p.subtitle{margin-top:.25em;margin-bottom:.5em}code a:any-link{color:inherit;text-decoration-color:#6c757d}/*! dark */div.observablehq table thead tr th{background-color:var(--bs-body-bg)}input,button,select,optgroup,textarea{background-color:var(--bs-body-bg)}.code-annotated .code-copy-button{margin-right:1.25em;margin-top:0;padding-bottom:0;padding-top:3px}.code-annotation-gutter-bg{background-color:#272b30}.code-annotation-gutter{background-color:rgba(37,41,46,.65)}.code-annotation-gutter,.code-annotation-gutter-bg{height:100%;width:calc(20px + .5em);position:absolute;top:0;right:0}dl.code-annotation-container-grid dt{margin-right:1em;margin-top:.25rem}dl.code-annotation-container-grid dt{font-family:var(--bs-font-monospace);color:#9ca3aa;border:solid #9ca3aa 1px;border-radius:50%;height:22px;width:22px;line-height:22px;font-size:11px;text-align:center;vertical-align:middle;text-decoration:none}dl.code-annotation-container-grid dt[data-target-cell]{cursor:pointer}dl.code-annotation-container-grid dt[data-target-cell].code-annotation-active{color:#272b30;border:solid #aaa 1px;background-color:#aaa}pre.code-annotation-code{padding-top:0;padding-bottom:0}pre.code-annotation-code code{z-index:3}#code-annotation-line-highlight-gutter{width:100%;border-top:solid rgba(170,170,170,.2666666667) 1px;border-bottom:solid rgba(170,170,170,.2666666667) 1px;z-index:2;background-color:rgba(170,170,170,.1333333333)}#code-annotation-line-highlight{margin-left:-4em;width:calc(100% + 4em);border-top:solid rgba(170,170,170,.2666666667) 1px;border-bottom:solid rgba(170,170,170,.2666666667) 1px;z-index:2;background-color:rgba(170,170,170,.1333333333)}code.sourceCode .code-annotation-anchor.code-annotation-active{background-color:var(--quarto-hl-normal-color, #aaaaaa);border:solid var(--quarto-hl-normal-color, #aaaaaa) 1px;color:#25292e;font-weight:bolder}code.sourceCode .code-annotation-anchor{font-family:var(--bs-font-monospace);color:var(--quarto-hl-co-color);border:solid var(--quarto-hl-co-color) 1px;border-radius:50%;height:18px;width:18px;font-size:9px;margin-top:2px}code.sourceCode button.code-annotation-anchor{padding:2px}code.sourceCode a.code-annotation-anchor{line-height:18px;text-align:center;vertical-align:middle;cursor:default;text-decoration:none}@media print{.page-columns .column-screen-inset{grid-column:page-start-inset/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset table{background:#272b30}.page-columns .column-screen-inset-left{grid-column:page-start-inset/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-left table{background:#272b30}.page-columns .column-screen-inset-right{grid-column:body-content-start/page-end-inset;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-inset-right table{background:#272b30}.page-columns .column-screen{grid-column:page-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen table{background:#272b30}.page-columns .column-screen-left{grid-column:page-start/body-content-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-left table{background:#272b30}.page-columns .column-screen-right{grid-column:body-content-start/page-end;z-index:998;transform:translate3d(0, 0, 0)}.page-columns .column-screen-right table{background:#272b30}.page-columns .column-screen-inset-shaded{grid-column:page-start-inset/page-end-inset;padding:1em;background:#d6dade;z-index:998;transform:translate3d(0, 0, 0);margin-bottom:1em}}.quarto-video{margin-bottom:1em}.table>thead{border-top-width:0}.table>:not(caption)>*:not(:last-child)>*{border-bottom-color:#e6e9eb;border-bottom-style:solid;border-bottom-width:1px}.table>:not(:first-child){border-top:1px solid #d6dade;border-bottom:1px solid inherit}.table tbody{border-bottom-color:#d6dade}a.external:after{display:inline-block;height:.75rem;width:.75rem;margin-bottom:.15em;margin-left:.25em;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:.75rem .75rem}div.sourceCode code a.external:after{content:none}a.external:after:hover{cursor:pointer}.quarto-ext-icon{display:inline-block;font-size:.75em;padding-left:.3em}.code-with-filename .code-with-filename-file{margin-bottom:0;padding-bottom:2px;padding-top:2px;padding-left:.7em;border:var(--quarto-border-width) solid var(--quarto-border-color);border-radius:var(--quarto-border-radius);border-bottom:0;border-bottom-left-radius:0%;border-bottom-right-radius:0%}.code-with-filename div.sourceCode,.reveal .code-with-filename div.sourceCode{margin-top:0;border-top-left-radius:0%;border-top-right-radius:0%}.code-with-filename .code-with-filename-file pre{margin-bottom:0}.code-with-filename .code-with-filename-file,.code-with-filename .code-with-filename-file pre{background-color:rgba(219,219,219,.8)}.quarto-dark .code-with-filename .code-with-filename-file,.quarto-dark .code-with-filename .code-with-filename-file pre{background-color:#555}.code-with-filename .code-with-filename-file strong{font-weight:400}.quarto-title-banner{margin-bottom:1em;color:#d8dadb;background:#3e444a}.quarto-title-banner .code-tools-button{color:#adaeaf}.quarto-title-banner .code-tools-button:hover{color:#d8dadb}.quarto-title-banner .code-tools-button>.bi::before{background-image:url('data:image/svg+xml,')}.quarto-title-banner .code-tools-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}.quarto-title-banner .quarto-title .title{font-weight:600}.quarto-title-banner .quarto-categories{margin-top:.75em}@media(min-width: 992px){.quarto-title-banner{padding-top:2.5em;padding-bottom:2.5em}}@media(max-width: 991.98px){.quarto-title-banner{padding-top:1em;padding-bottom:1em}}main.quarto-banner-title-block>section:first-child>h2,main.quarto-banner-title-block>section:first-child>.h2,main.quarto-banner-title-block>section:first-child>h3,main.quarto-banner-title-block>section:first-child>.h3,main.quarto-banner-title-block>section:first-child>h4,main.quarto-banner-title-block>section:first-child>.h4{margin-top:0}.quarto-title .quarto-categories{display:flex;flex-wrap:wrap;row-gap:.5em;column-gap:.4em;padding-bottom:.5em;margin-top:.75em}.quarto-title .quarto-categories .quarto-category{padding:.25em .75em;font-size:.65em;text-transform:uppercase;border:solid 1px;border-radius:.25rem;opacity:.6}.quarto-title .quarto-categories .quarto-category a{color:inherit}#title-block-header.quarto-title-block.default .quarto-title-meta{display:grid;grid-template-columns:repeat(2, 1fr)}#title-block-header.quarto-title-block.default .quarto-title .title{margin-bottom:0}#title-block-header.quarto-title-block.default .quarto-title-author-orcid img{margin-top:-5px}#title-block-header.quarto-title-block.default .quarto-description p:last-of-type{margin-bottom:0}#title-block-header.quarto-title-block.default .quarto-title-meta-contents p,#title-block-header.quarto-title-block.default .quarto-title-authors p,#title-block-header.quarto-title-block.default .quarto-title-affiliations p{margin-bottom:.1em}#title-block-header.quarto-title-block.default .quarto-title-meta-heading{text-transform:uppercase;margin-top:1em;font-size:.8em;opacity:.8;font-weight:400}#title-block-header.quarto-title-block.default .quarto-title-meta-contents{font-size:.9em}#title-block-header.quarto-title-block.default .quarto-title-meta-contents a{color:#adb5bd}#title-block-header.quarto-title-block.default .quarto-title-meta-contents p.affiliation:last-of-type{margin-bottom:.7em}#title-block-header.quarto-title-block.default p.affiliation{margin-bottom:.1em}#title-block-header.quarto-title-block.default .description,#title-block-header.quarto-title-block.default .abstract{margin-top:0}#title-block-header.quarto-title-block.default .description>p,#title-block-header.quarto-title-block.default .abstract>p{font-size:.9em}#title-block-header.quarto-title-block.default .description>p:last-of-type,#title-block-header.quarto-title-block.default .abstract>p:last-of-type{margin-bottom:0}#title-block-header.quarto-title-block.default .description .abstract-title,#title-block-header.quarto-title-block.default .abstract .abstract-title{margin-top:1em;text-transform:uppercase;font-size:.8em;opacity:.8;font-weight:400}#title-block-header.quarto-title-block.default .quarto-title-meta-author{display:grid;grid-template-columns:1fr 1fr}.quarto-title-tools-only{display:flex;justify-content:right}.navbar{border:1px solid rgba(0,0,0,.6);text-shadow:1px 1px 1px rgba(0,0,0,.3)}.navbar .container{padding:0}.navbar .navbar-toggler{border-color:rgba(0,0,0,.6)}.navbar-fixed-top{border-width:0 0 1px}.navbar-fixed-bottom{border-width:1px 0 0}.navbar .nav-link{padding:1rem;border-left:1px solid rgba(255,255,255,.1);border-right:1px solid rgba(0,0,0,.2)}.navbar .nav-link:hover,.navbar .nav-link:focus{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;border-left:1px solid rgba(0,0,0,.2)}.navbar-brand{padding:.75rem 1rem calc(24px - 0.75rem);margin-right:0;border-right:1px solid rgba(0,0,0,.2)}.navbar .nav-item.active .nav-link{background-color:rgba(0,0,0,.3);border-left:1px solid rgba(0,0,0,.2)}.navbar-nav .nav-item+.nav-item{margin-left:0}.navbar.bg-light{text-shadow:1px 1px 1px rgba(0,0,0,.1)}.navbar.bg-light .nav-link:hover,.navbar.bg-light .nav-link:focus{background-image:linear-gradient(#6f7479, #797f84 40%, #80868c);filter:none;border-left:1px solid rgba(0,0,0,.2)}@media(max-width: 576px){.navbar-expand-sm .navbar-brand,.navbar-expand-sm .nav-link{border:none !important}}@media(max-width: 768px){.navbar-expand-md .navbar-brand,.navbar-expand-md .nav-link{border:none !important}}@media(max-width: 992px){.navbar-expand-lg .navbar-brand,.navbar-expand-lg .nav-link{border:none !important}}.btn{border-color:rgba(0,0,0,.6);text-shadow:1px 1px 1px rgba(0,0,0,.3)}.btn:not([disabled]):not(.disabled).active,.btn.disabled{border-color:rgba(0,0,0,.6);box-shadow:none}.btn:hover,.btn:focus,.btn:not([disabled]):not(.disabled):active,.btn:not([disabled]):not(.disabled):active:hover,.btn:not([disabled]):not(.disabled).active:hover{border-color:rgba(0,0,0,.6)}.btn-primary{background-image:linear-gradient(#555a60, #3e444a 60%, #393f44);filter:none}.btn-primary:not([disabled]):not(.disabled):hover,.btn-primary:not([disabled]):not(.disabled):focus,.btn-primary:not([disabled]):not(.disabled):active:hover,.btn-primary:not([disabled]):not(.disabled).active:hover{background-image:linear-gradient(#282c2f, #2b3034 40%, #2e3237);filter:none}.btn-default{background-image:linear-gradient(#7e868d, #6c757d 60%, #636c73);filter:none;color:#fff}.btn-default:not([disabled]):not(.disabled):hover,.btn-default:not([disabled]):not(.disabled):focus,.btn-default:not([disabled]):not(.disabled):active,.btn-default:not([disabled]):not(.disabled).active{background-image:linear-gradient(#454b50, #4c5258 40%, #50575d);filter:none}.btn-secondary{background-image:linear-gradient(#7e868d, #6c757d 60%, #636c73);filter:none}.btn-secondary:not([disabled]):not(.disabled):hover,.btn-secondary:not([disabled]):not(.disabled):focus,.btn-secondary:not([disabled]):not(.disabled):active,.btn-secondary:not([disabled]):not(.disabled).active{background-image:linear-gradient(#454b50, #4c5258 40%, #50575d);filter:none}.btn-success{background-image:linear-gradient(#75cb75, #62c462 60%, #5ab45a);filter:none;color:#fff}.btn-success:not([disabled]):not(.disabled):hover,.btn-success:not([disabled]):not(.disabled):focus,.btn-success:not([disabled]):not(.disabled):active,.btn-success:not([disabled]):not(.disabled).active{background-image:linear-gradient(#3f7d3f, #458945 40%, #499149);filter:none}.btn-info{background-image:linear-gradient(#6fc8e2, #5bc0de 60%, #54b1cc);filter:none;color:#fff}.btn-info:not([disabled]):not(.disabled):hover,.btn-info:not([disabled]):not(.disabled):focus,.btn-info:not([disabled]):not(.disabled):active,.btn-info:not([disabled]):not(.disabled).active{background-image:linear-gradient(#3a7b8e, #40869b 40%, #438ea4);filter:none}.btn-warning{background-image:linear-gradient(#f9a124, #f89406 60%, #e48806);filter:none;color:#fff}.btn-warning:not([disabled]):not(.disabled):hover,.btn-warning:not([disabled]):not(.disabled):focus,.btn-warning:not([disabled]):not(.disabled):active,.btn-warning:not([disabled]):not(.disabled).active{background-image:linear-gradient(#9f5f04, #ae6804 40%, #b86e04);filter:none}.btn-danger{background-image:linear-gradient(#f0726f, #ee5f5b 60%, #db5754);filter:none}.btn-danger:not([disabled]):not(.disabled):hover,.btn-danger:not([disabled]):not(.disabled):focus,.btn-danger:not([disabled]):not(.disabled):active,.btn-danger:not([disabled]):not(.disabled).active{background-image:linear-gradient(#983d3a, #a74340 40%, #b04643);filter:none}.btn-outline-primary{color:#fff}.btn-link,.btn-link:hover{border-color:rgba(0,0,0,0)}.btn-group .btn.active,.btn-group-vertical .btn.active{border-color:rgba(0,0,0,.6)}h1,.h1,h2,.h2,h3,.h3,h4,.h4,h5,.h5,h6,.h6{text-shadow:-1px -1px 0 rgba(0,0,0,.3)}legend{color:#fff}.input-group-addon{background-image:linear-gradient(#7e868d, #6c757d 60%, #636c73);filter:none;text-shadow:1px 1px 1px rgba(0,0,0,.3);color:#fff}.nav-tabs .nav-link{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;border:1px solid rgba(0,0,0,.6)}.nav-tabs .nav-link:not([disabled]):not(.disabled):hover,.nav-tabs .nav-link:not([disabled]):not(.disabled):focus,.nav-tabs .nav-link:not([disabled]):not(.disabled):active,.nav-tabs .nav-link:not([disabled]):not(.disabled).active{background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none}.nav-tabs .nav-link.disabled{border:1px solid rgba(0,0,0,.6)}.nav-tabs .nav-link,.nav-tabs .nav-link:hover{color:#fff}.nav-pills .nav-link{background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none;border:1px solid rgba(0,0,0,.6);text-shadow:1px 1px 1px rgba(0,0,0,.3);color:#fff}.nav-pills .nav-link:hover{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;border:1px solid rgba(0,0,0,.6)}.nav-pills .nav-link.active,.nav-pills .nav-link:hover{background-color:rgba(0,0,0,0);background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;border:1px solid rgba(0,0,0,.6)}.nav-pills .nav-link.disabled,.nav-pills .nav-link.disabled:hover{background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none;color:#6c757d}.pagination .page-link{text-shadow:1px 1px 1px rgba(0,0,0,.3);background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none}.pagination .page-link:hover{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none;text-decoration:none}.pagination .page-item.active .page-link{background-image:linear-gradient(#191c1f, #1b1e22 40%, #1d2024);filter:none}.pagination .page-item.disabled .page-link{background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none}.breadcrumb{border:1px solid rgba(0,0,0,.6);text-shadow:1px 1px 1px rgba(0,0,0,.3);background-color:rgba(0,0,0,0);background-image:linear-gradient(#414449, #272b30 60%, #24282c);filter:none}.breadcrumb a,.breadcrumb a:hover{color:#fff}.alert{border:none;color:#fff}.alert a,.alert .alert-link{color:#fff;text-decoration:underline}.alert-default{background-color:#6c757d}.alert-primary{background-color:#3e444a}.alert-secondary{background-color:#6c757d}.alert-success{background-color:#62c462}.alert-info{background-color:#5bc0de}.alert-warning{background-color:#f89406}.alert-danger{background-color:#ee5f5b}.alert-light{background-color:#d6dade}.alert-dark{background-color:#272b30}.alert-light,.alert-light a:not(.btn),.alert-light .alert-link{color:#272b30}.badge.bg-light{color:#272b30}.list-group-item-action:hover{background-color:#23272b}/*# sourceMappingURL=9161419e6f82ea4435380a70856fa72b.css.map */
diff --git a/sitemap.xml b/sitemap.xml
index 1491170..56efa68 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -1,91 +1,91 @@
- https://dfe-analytical-services.github.io/analysts-guide/writing-visualising/dashboards.html
- 2023-09-30T15:01:14.536Z
-
-
- https://dfe-analytical-services.github.io/analysts-guide/statistics-production/user-eng.html
- 2023-09-30T15:01:08.448Z
+ https://dfe-analytical-services.github.io/analysts-guide/CONTRIBUTING.html
+ 2023-09-30T15:24:12.928Z
- https://dfe-analytical-services.github.io/analysts-guide/statistics-production/ud.html
- 2023-09-30T15:01:05.541Z
+ https://dfe-analytical-services.github.io/analysts-guide/statistics-production/examples.html
+ 2023-09-30T15:23:47.619Z
- https://dfe-analytical-services.github.io/analysts-guide/statistics-production/pub.html
- 2023-09-30T15:00:50.229Z
+ https://dfe-analytical-services.github.io/analysts-guide/statistics-production/ees.html
+ 2023-09-30T15:23:46.055Z
- https://dfe-analytical-services.github.io/analysts-guide/statistics-production/embedded-charts.html
- 2023-09-30T15:00:47.442Z
+ https://dfe-analytical-services.github.io/analysts-guide/statistics-production/user-analytics.html
+ 2023-09-30T15:23:41.783Z
- https://dfe-analytical-services.github.io/analysts-guide/RAP/rap-support.html
- 2023-09-30T15:00:42.380Z
+ https://dfe-analytical-services.github.io/analysts-guide/statistics-production/ud.html
+ 2023-09-30T15:23:39.927Z
- https://dfe-analytical-services.github.io/analysts-guide/RAP/rap-faq.html
- 2023-09-30T15:00:29.460Z
+ https://dfe-analytical-services.github.io/analysts-guide/learning-development/r.html
+ 2023-09-30T15:23:10.302Zhttps://dfe-analytical-services.github.io/analysts-guide/learning-development/sql.html
- 2023-09-30T15:00:26.856Z
+ 2023-09-30T15:23:08.334Zhttps://dfe-analytical-services.github.io/analysts-guide/learning-development/python.html
- 2023-09-30T15:00:23.674Z
+ 2023-09-30T15:23:06.382Z
- https://dfe-analytical-services.github.io/analysts-guide/learning-development/git.html
- 2023-09-30T15:00:21.512Z
+ https://dfe-analytical-services.github.io/analysts-guide/RAP/rap-faq.html
+ 2023-09-30T15:23:05.170Z
- https://dfe-analytical-services.github.io/analysts-guide/CONTRIBUTING.html
- 2023-09-30T15:00:06.532Z
+ https://dfe-analytical-services.github.io/analysts-guide/RAP/rap-statistics.html
+ 2023-09-30T15:23:03.462Z
- https://dfe-analytical-services.github.io/analysts-guide/CODE_OF_CONDUCT.html
- 2023-09-30T15:00:05.238Z
+ https://dfe-analytical-services.github.io/analysts-guide/writing-visualising/cd.html
+ 2023-09-30T15:22:30.873Z
- https://dfe-analytical-services.github.io/analysts-guide/index.html
- 2023-09-30T15:00:19.768Z
+ https://dfe-analytical-services.github.io/analysts-guide/writing-visualising/dashboards.html
+ 2023-09-30T15:22:35.813Z
- https://dfe-analytical-services.github.io/analysts-guide/learning-development/learning-support.html
- 2023-09-30T15:00:22.720Z
+ https://dfe-analytical-services.github.io/analysts-guide/RAP/rap-expectations.html
+ 2023-09-30T15:23:04.138Z
- https://dfe-analytical-services.github.io/analysts-guide/learning-development/r.html
- 2023-09-30T15:00:25.556Z
+ https://dfe-analytical-services.github.io/analysts-guide/RAP/rap-support.html
+ 2023-09-30T15:23:05.782Z
- https://dfe-analytical-services.github.io/analysts-guide/RAP/rap-expectations.html
- 2023-09-30T15:00:28.017Z
+ https://dfe-analytical-services.github.io/analysts-guide/learning-development/git.html
+ 2023-09-30T15:23:07.574Z
- https://dfe-analytical-services.github.io/analysts-guide/RAP/rap-statistics.html
- 2023-09-30T15:00:41.240Z
+ https://dfe-analytical-services.github.io/analysts-guide/learning-development/learning-support.html
+ 2023-09-30T15:23:09.010Z
- https://dfe-analytical-services.github.io/analysts-guide/statistics-production/ees.html
- 2023-09-30T15:00:46.178Z
+ https://dfe-analytical-services.github.io/analysts-guide/statistics-production/stats_tools.html
+ 2023-09-30T15:23:10.874Z
- https://dfe-analytical-services.github.io/analysts-guide/statistics-production/examples.html
- 2023-09-30T15:00:48.837Z
+ https://dfe-analytical-services.github.io/analysts-guide/statistics-production/user-eng.html
+ 2023-09-30T15:23:41.187Z
- https://dfe-analytical-services.github.io/analysts-guide/statistics-production/stats_tools.html
- 2023-09-30T15:00:51.220Z
+ https://dfe-analytical-services.github.io/analysts-guide/statistics-production/pub.html
+ 2023-09-30T15:23:42.651Z
- https://dfe-analytical-services.github.io/analysts-guide/statistics-production/user-analytics.html
- 2023-09-30T15:01:06.718Z
+ https://dfe-analytical-services.github.io/analysts-guide/statistics-production/embedded-charts.html
+ 2023-09-30T15:23:46.823Z
- https://dfe-analytical-services.github.io/analysts-guide/writing-visualising/cd.html
- 2023-09-30T15:01:10.916Z
+ https://dfe-analytical-services.github.io/analysts-guide/CODE_OF_CONDUCT.html
+ 2023-09-30T15:23:48.275Z
+
+
+ https://dfe-analytical-services.github.io/analysts-guide/index.html
+ 2023-09-30T15:24:12.336Z
diff --git a/statistics-production/ees.html b/statistics-production/ees.html
index 8f07b3b..0265df4 100644
--- a/statistics-production/ees.html
+++ b/statistics-production/ees.html
@@ -2,7 +2,7 @@
-
+
@@ -29,6 +29,8 @@
+
+
@@ -100,7 +102,7 @@
-
+
