diff --git a/pr-previews/462/search.json b/pr-previews/462/search.json index 7d2ae14ec..a9f547ca2 100644 --- a/pr-previews/462/search.json +++ b/pr-previews/462/search.json @@ -723,7 +723,7 @@ "href": "tutorials/03-bayesian-neural-network/index.html", "title": "Bayesian Neural Networks", "section": "", - "text": "In this tutorial, we demonstrate how one can implement a Bayesian Neural Network using a combination of Turing and Flux, a suite of machine learning tools. We will use Flux to specify the neural network’s layers and Turing to implement the probabilistic inference, with the goal of implementing a classification algorithm.\nWe will begin with importing the relevant libraries.\nusing Turing\nusing FillArrays\nusing Lux\nusing Plots\nusing ReverseDiff\nusing Functors\n\nusing LinearAlgebra\nusing Random\nOur goal here is to use a Bayesian neural network to classify points in an artificial dataset. The code below generates data points arranged in a box-like pattern and displays a graph of the dataset we will be working with.\n# Number of points to generate\nN = 80\nM = round(Int, N / 4)\nrng = Random.default_rng()\nRandom.seed!(rng, 1234)\n\n# Generate artificial data\nx1s = rand(rng, Float32, M) * 4.5f0;\nx2s = rand(rng, Float32, M) * 4.5f0;\nxt1s = Array([[x1s[i] + 0.5f0; x2s[i] + 0.5f0] for i in 1:M])\nx1s = rand(rng, Float32, M) * 4.5f0;\nx2s = rand(rng, Float32, M) * 4.5f0;\nappend!(xt1s, Array([[x1s[i] - 5.0f0; x2s[i] - 5.0f0] for i in 1:M]))\n\nx1s = rand(rng, Float32, M) * 4.5f0;\nx2s = rand(rng, Float32, M) * 4.5f0;\nxt0s = Array([[x1s[i] + 0.5f0; x2s[i] - 5.0f0] for i in 1:M])\nx1s = rand(rng, Float32, M) * 4.5f0;\nx2s = rand(rng, Float32, M) * 4.5f0;\nappend!(xt0s, Array([[x1s[i] - 5.0f0; x2s[i] + 0.5f0] for i in 1:M]))\n\n# Store all the data for later\nxs = [xt1s; xt0s]\nts = [ones(2 * M); zeros(2 * M)]\n\n# Plot data points.\nfunction plot_data()\n x1 = map(e -> e[1], xt1s)\n y1 = map(e -> e[2], xt1s)\n x2 = map(e -> e[1], xt0s)\n y2 = map(e -> e[2], xt0s)\n\n Plots.scatter(x1, y1; color=\"red\", clim=(0, 1))\n return Plots.scatter!(x2, y2; color=\"blue\", clim=(0, 1))\nend\n\nplot_data()", + "text": "In this tutorial, we demonstrate how one can implement a Bayesian Neural Network using a combination of Turing and Flux, a suite of machine learning tools. We will use Flux to specify the neural network’s layers and Turing to implement the probabilistic inference, with the goal of implementing a classification algorithm.\nWe will begin with importing the relevant libraries.\nusing Turing\nusing FillArrays\nusing Lux\nusing Plots\nusing Tracker\nusing Functors\n\nusing LinearAlgebra\nusing Random\nOur goal here is to use a Bayesian neural network to classify points in an artificial dataset. The code below generates data points arranged in a box-like pattern and displays a graph of the dataset we will be working with.\n# Number of points to generate\nN = 80\nM = round(Int, N / 4)\nrng = Random.default_rng()\nRandom.seed!(rng, 1234)\n\n# Generate artificial data\nx1s = rand(rng, Float32, M) * 4.5f0;\nx2s = rand(rng, Float32, M) * 4.5f0;\nxt1s = Array([[x1s[i] + 0.5f0; x2s[i] + 0.5f0] for i in 1:M])\nx1s = rand(rng, Float32, M) * 4.5f0;\nx2s = rand(rng, Float32, M) * 4.5f0;\nappend!(xt1s, Array([[x1s[i] - 5.0f0; x2s[i] - 5.0f0] for i in 1:M]))\n\nx1s = rand(rng, Float32, M) * 4.5f0;\nx2s = rand(rng, Float32, M) * 4.5f0;\nxt0s = Array([[x1s[i] + 0.5f0; x2s[i] - 5.0f0] for i in 1:M])\nx1s = rand(rng, Float32, M) * 4.5f0;\nx2s = rand(rng, Float32, M) * 4.5f0;\nappend!(xt0s, Array([[x1s[i] - 5.0f0; x2s[i] + 0.5f0] for i in 1:M]))\n\n# Store all the data for later\nxs = [xt1s; xt0s]\nts = [ones(2 * M); zeros(2 * M)]\n\n# Plot data points.\nfunction plot_data()\n x1 = map(e -> e[1], xt1s)\n y1 = map(e -> e[2], xt1s)\n x2 = map(e -> e[1], xt0s)\n y2 = map(e -> e[2], xt0s)\n\n Plots.scatter(x1, y1; color=\"red\", clim=(0, 1))\n return Plots.scatter!(x2, y2; color=\"blue\", clim=(0, 1))\nend\n\nplot_data()", "crumbs": [ "Documentation", "Using Turing - Tutorials", @@ -735,7 +735,7 @@ "href": "tutorials/03-bayesian-neural-network/index.html#building-a-neural-network", "title": "Bayesian Neural Networks", "section": "Building a Neural Network", - "text": "Building a Neural Network\nThe next step is to define a feedforward neural network where we express our parameters as distributions, and not single points as with traditional neural networks. For this we will use Dense to define liner layers and compose them via Chain, both are neural network primitives from Lux. The network nn_initial we created has two hidden layers with tanh activations and one output layer with sigmoid (σ) activation, as shown below.\n\n\n\n\n\n\n\nG\n\nInput layer                   Hidden layers                  Output layer\n\ncluster_input\n\n\n\ncluster_hidden1\n\n\n\ncluster_hidden2\n\n\n\ncluster_output\n\n\n\n\ninput1\n\n\n\n\nhidden11\n\n\n\n\ninput1--hidden11\n\n\n\n\nhidden12\n\n\n\n\ninput1--hidden12\n\n\n\n\nhidden13\n\n\n\n\ninput1--hidden13\n\n\n\n\ninput2\n\n\n\n\ninput2--hidden11\n\n\n\n\ninput2--hidden12\n\n\n\n\ninput2--hidden13\n\n\n\n\nhidden21\n\n\n\n\nhidden11--hidden21\n\n\n\n\nhidden22\n\n\n\n\nhidden11--hidden22\n\n\n\n\nhidden12--hidden21\n\n\n\n\nhidden12--hidden22\n\n\n\n\nhidden13--hidden21\n\n\n\n\nhidden13--hidden22\n\n\n\n\noutput1\n\n\n\n\nhidden21--output1\n\n\n\n\nhidden22--output1\n\n\n\n\n\n\n\n\n\nThe nn_initial is an instance that acts as a function and can take data as inputs and output predictions. We will define distributions on the neural network parameters. \n\n# Construct a neural network using Lux\nnn_initial = Chain(Dense(2 => 3, tanh), Dense(3 => 2, tanh), Dense(2 => 1, σ))\n\n# Initialize the model weights and state\nps, st = Lux.setup(rng, nn_initial)\n\nLux.parameterlength(nn_initial) # number of paraemters in NN\n\n20\n\n\nThe probabilistic model specification below creates a parameters variable, which has IID normal variables. The parameters vector represents all parameters of our neural net (weights and biases).\n\n# Create a regularization term and a Gaussian prior variance term.\nalpha = 0.09\nsigma = sqrt(1.0 / alpha)\n\n3.3333333333333335\n\n\nConstruct named tuple from a sampled parameter vector. We could also use ComponentArrays here and simply broadcast to avoid doing this. But let’s do it this way to avoid dependencies.\n\nfunction vector_to_parameters(ps_new::AbstractVector, ps::NamedTuple)\n @assert length(ps_new) == Lux.parameterlength(ps)\n i = 1\n function get_ps(x)\n z = reshape(view(ps_new, i:(i + length(x) - 1)), size(x))\n i += length(x)\n return z\n end\n return fmap(get_ps, ps)\nend\n\nvector_to_parameters (generic function with 1 method)\n\n\nTo interface with external libraries it is often desirable to use the StatefulLuxLayer to automatically handle the neural network states.\n\nconst nn = StatefulLuxLayer(nn_initial, st)\n\n# Specify the probabilistic model.\n@model function bayes_nn(xs, ts; sigma = sigma, ps = ps, nn = nn)\n # Sample the parameters\n nparameters = Lux.parameterlength(nn_initial)\n parameters ~ MvNormal(zeros(nparameters), Diagonal(abs2.(sigma .* ones(nparameters))))\n\n # Forward NN to make predictions\n preds = Lux.apply(nn, xs, vector_to_parameters(parameters, ps))\n\n # Observe each prediction.\n for i in eachindex(ts)\n ts[i] ~ Bernoulli(preds[i])\n end\nend\n\nbayes_nn (generic function with 2 methods)\n\n\nInference can now be performed by calling sample. We use the NUTS Hamiltonian Monte Carlo sampler here.\n\nsetprogress!(false)\n\n\n# Perform inference.\nN = 500\nch = sample(bayes_nn(reduce(hcat, xs), ts), NUTS(; adtype=AutoReverseDiff()), N);\n\n┌ Warning: Lux.apply(m::Lux.AbstractExplicitLayer, x::AbstractArray{<:ReverseDiff.TrackedReal}, ps, st) input was corrected to Lux.apply(m::Lux.AbstractExplicitLayer, x::ReverseDiff.TrackedArray}, ps, st).\n│ \n│ 1. If this was not the desired behavior overload the dispatch on `m`.\n│ \n│ 2. This might have performance implications. Check which layer was causing this problem using `Lux.Experimental.@debug_mode`.\n└ @ LuxReverseDiffExt ~/.julia/packages/Lux/PsbZF/ext/LuxReverseDiffExt.jl:25\n┌ Info: Found initial step size\n└ ϵ = 0.8\n\n\nNow we extract the parameter samples from the sampled chain as θ (this is of size 5000 x 20 where 5000 is the number of iterations and 20 is the number of parameters). We’ll use these primarily to determine how good our model’s classifier is.\n\n# Extract all weight and bias parameters.\nθ = MCMCChains.group(ch, :parameters).value;", + "text": "Building a Neural Network\nThe next step is to define a feedforward neural network where we express our parameters as distributions, and not single points as with traditional neural networks. For this we will use Dense to define liner layers and compose them via Chain, both are neural network primitives from Lux. The network nn_initial we created has two hidden layers with tanh activations and one output layer with sigmoid (σ) activation, as shown below.\n\n\n\n\n\n\n\nG\n\nInput layer                   Hidden layers                  Output layer\n\ncluster_input\n\n\n\ncluster_hidden1\n\n\n\ncluster_hidden2\n\n\n\ncluster_output\n\n\n\n\ninput1\n\n\n\n\nhidden11\n\n\n\n\ninput1--hidden11\n\n\n\n\nhidden12\n\n\n\n\ninput1--hidden12\n\n\n\n\nhidden13\n\n\n\n\ninput1--hidden13\n\n\n\n\ninput2\n\n\n\n\ninput2--hidden11\n\n\n\n\ninput2--hidden12\n\n\n\n\ninput2--hidden13\n\n\n\n\nhidden21\n\n\n\n\nhidden11--hidden21\n\n\n\n\nhidden22\n\n\n\n\nhidden11--hidden22\n\n\n\n\nhidden12--hidden21\n\n\n\n\nhidden12--hidden22\n\n\n\n\nhidden13--hidden21\n\n\n\n\nhidden13--hidden22\n\n\n\n\noutput1\n\n\n\n\nhidden21--output1\n\n\n\n\nhidden22--output1\n\n\n\n\n\n\n\n\n\nThe nn_initial is an instance that acts as a function and can take data as inputs and output predictions. We will define distributions on the neural network parameters. \n\n# Construct a neural network using Lux\nnn_initial = Chain(Dense(2 => 3, tanh), Dense(3 => 2, tanh), Dense(2 => 1, σ))\n\n# Initialize the model weights and state\nps, st = Lux.setup(rng, nn_initial)\n\nLux.parameterlength(nn_initial) # number of paraemters in NN\n\n20\n\n\nThe probabilistic model specification below creates a parameters variable, which has IID normal variables. The parameters vector represents all parameters of our neural net (weights and biases).\n\n# Create a regularization term and a Gaussian prior variance term.\nalpha = 0.09\nsigma = sqrt(1.0 / alpha)\n\n3.3333333333333335\n\n\nConstruct named tuple from a sampled parameter vector. We could also use ComponentArrays here and simply broadcast to avoid doing this. But let’s do it this way to avoid dependencies.\n\nfunction vector_to_parameters(ps_new::AbstractVector, ps::NamedTuple)\n @assert length(ps_new) == Lux.parameterlength(ps)\n i = 1\n function get_ps(x)\n z = reshape(view(ps_new, i:(i + length(x) - 1)), size(x))\n i += length(x)\n return z\n end\n return fmap(get_ps, ps)\nend\n\nvector_to_parameters (generic function with 1 method)\n\n\nTo interface with external libraries it is often desirable to use the StatefulLuxLayer to automatically handle the neural network states.\n\nconst nn = StatefulLuxLayer(nn_initial, st)\n\n# Specify the probabilistic model.\n@model function bayes_nn(xs, ts; sigma = sigma, ps = ps, nn = nn)\n # Sample the parameters\n nparameters = Lux.parameterlength(nn_initial)\n parameters ~ MvNormal(zeros(nparameters), Diagonal(abs2.(sigma .* ones(nparameters))))\n\n # Forward NN to make predictions\n preds = Lux.apply(nn, xs, vector_to_parameters(parameters, ps))\n\n # Observe each prediction.\n for i in eachindex(ts)\n ts[i] ~ Bernoulli(preds[i])\n end\nend\n\nbayes_nn (generic function with 2 methods)\n\n\nInference can now be performed by calling sample. We use the NUTS Hamiltonian Monte Carlo sampler here.\n\nsetprogress!(false)\n\n\n# Perform inference.\nN = 2_000\nch = sample(bayes_nn(reduce(hcat, xs), ts), NUTS(; adtype=AutoTracker()), N);\n\n┌ Info: Found initial step size\n└ ϵ = 0.4\n\n\nNow we extract the parameter samples from the sampled chain as θ (this is of size 5000 x 20 where 5000 is the number of iterations and 20 is the number of parameters). We’ll use these primarily to determine how good our model’s classifier is.\n\n# Extract all weight and bias parameters.\nθ = MCMCChains.group(ch, :parameters).value;", "crumbs": [ "Documentation", "Using Turing - Tutorials", @@ -747,7 +747,7 @@ "href": "tutorials/03-bayesian-neural-network/index.html#prediction-visualization", "title": "Bayesian Neural Networks", "section": "Prediction Visualization", - "text": "Prediction Visualization\nWe can use MAP estimation to classify our population by using the set of weights that provided the highest log posterior.\n\n# A helper to run the nn through data `x` using parameters `θ`\nnn_forward(x, θ) = nn(x, vector_to_parameters(θ, ps))\n\n# Plot the data we have.\nfig = plot_data()\n\n# Find the index that provided the highest log posterior in the chain.\n_, i = findmax(ch[:lp])\n\n# Extract the max row value from i.\ni = i.I[1]\n\n# Plot the posterior distribution with a contour plot\nx1_range = collect(range(-6; stop=6, length=25))\nx2_range = collect(range(-6; stop=6, length=25))\nZ = [nn_forward([x1, x2], θ[i, :])[1] for x1 in x1_range, x2 in x2_range]\ncontour!(x1_range, x2_range, Z; linewidth=3, colormap=:seaborn_bright)\nfig\n\n\n\n\n \n \n \n\n\n\n \n \n \n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nThe contour plot above shows that the MAP method is not too bad at classifying our data.\nNow we can visualize our predictions.\n\\[\np(\\tilde{x} | X, \\alpha) = \\int_{\\theta} p(\\tilde{x} | \\theta) p(\\theta | X, \\alpha) \\approx \\sum_{\\theta \\sim p(\\theta | X, \\alpha)}f_{\\theta}(\\tilde{x})\n\\]\nThe nn_predict function takes the average predicted value from a network parameterized by weights drawn from the MCMC chain.\n\n# Return the average predicted value across\n# multiple weights.\nfunction nn_predict(x, θ, num)\n num = min(num, size(θ, 1)) # make sure num does not exceed the number of samples\n return mean([first(nn_forward(x, view(θ, i, :))) for i in 1:10:num])\nend\n\nnn_predict (generic function with 1 method)\n\n\nNext, we use the nn_predict function to predict the value at a sample of points where the x1 and x2 coordinates range between -6 and 6. As we can see below, we still have a satisfactory fit to our data, and more importantly, we can also see where the neural network is uncertain about its predictions much easier—those regions between cluster boundaries.\n\n# Plot the average prediction.\nfig = plot_data()\n\nn_end = 1500\nx1_range = collect(range(-6; stop=6, length=25))\nx2_range = collect(range(-6; stop=6, length=25))\nZ = [nn_predict([x1, x2], θ, n_end)[1] for x1 in x1_range, x2 in x2_range]\ncontour!(x1_range, x2_range, Z; linewidth=3, colormap=:seaborn_bright)\nfig\n\n\n\n\n \n \n \n\n\n\n \n \n \n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nSuppose we are interested in how the predictive power of our Bayesian neural network evolved between samples. In that case, the following graph displays an animation of the contour plot generated from the network weights in samples 1 to 1,000.\n\n# Number of iterations to plot.\nn_end = 500\n\nanim = @gif for i in 1:n_end\n plot_data()\n Z = [nn_forward([x1, x2], θ[i, :])[1] for x1 in x1_range, x2 in x2_range]\n contour!(x1_range, x2_range, Z; title=\"Iteration $i\", clim=(0, 1))\nend every 5\n\n[ Info: Saved animation to /tmp/jl_56xULkjare.gif\n\n\n\n\n\nThis has been an introduction to the applications of Turing and Flux in defining Bayesian neural networks.", + "text": "Prediction Visualization\nWe can use MAP estimation to classify our population by using the set of weights that provided the highest log posterior.\n\n# A helper to run the nn through data `x` using parameters `θ`\nnn_forward(x, θ) = nn(x, vector_to_parameters(θ, ps))\n\n# Plot the data we have.\nfig = plot_data()\n\n# Find the index that provided the highest log posterior in the chain.\n_, i = findmax(ch[:lp])\n\n# Extract the max row value from i.\ni = i.I[1]\n\n# Plot the posterior distribution with a contour plot\nx1_range = collect(range(-6; stop=6, length=25))\nx2_range = collect(range(-6; stop=6, length=25))\nZ = [nn_forward([x1, x2], θ[i, :])[1] for x1 in x1_range, x2 in x2_range]\ncontour!(x1_range, x2_range, Z; linewidth=3, colormap=:seaborn_bright)\nfig\n\n\n\n\n \n \n \n\n\n\n \n \n \n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nThe contour plot above shows that the MAP method is not too bad at classifying our data.\nNow we can visualize our predictions.\n\\[\np(\\tilde{x} | X, \\alpha) = \\int_{\\theta} p(\\tilde{x} | \\theta) p(\\theta | X, \\alpha) \\approx \\sum_{\\theta \\sim p(\\theta | X, \\alpha)}f_{\\theta}(\\tilde{x})\n\\]\nThe nn_predict function takes the average predicted value from a network parameterized by weights drawn from the MCMC chain.\n\n# Return the average predicted value across\n# multiple weights.\nfunction nn_predict(x, θ, num)\n num = min(num, size(θ, 1)) # make sure num does not exceed the number of samples\n return mean([first(nn_forward(x, view(θ, i, :))) for i in 1:10:num])\nend\n\nnn_predict (generic function with 1 method)\n\n\nNext, we use the nn_predict function to predict the value at a sample of points where the x1 and x2 coordinates range between -6 and 6. As we can see below, we still have a satisfactory fit to our data, and more importantly, we can also see where the neural network is uncertain about its predictions much easier—those regions between cluster boundaries.\n\n# Plot the average prediction.\nfig = plot_data()\n\nn_end = 1500\nx1_range = collect(range(-6; stop=6, length=25))\nx2_range = collect(range(-6; stop=6, length=25))\nZ = [nn_predict([x1, x2], θ, n_end)[1] for x1 in x1_range, x2 in x2_range]\ncontour!(x1_range, x2_range, Z; linewidth=3, colormap=:seaborn_bright)\nfig\n\n\n\n\n \n \n \n\n\n\n \n \n \n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nSuppose we are interested in how the predictive power of our Bayesian neural network evolved between samples. In that case, the following graph displays an animation of the contour plot generated from the network weights in samples 1 to 1,000.\n\n# Number of iterations to plot.\nn_end = 500\n\nanim = @gif for i in 1:n_end\n plot_data()\n Z = [nn_forward([x1, x2], θ[i, :])[1] for x1 in x1_range, x2 in x2_range]\n contour!(x1_range, x2_range, Z; title=\"Iteration $i\", clim=(0, 1))\nend every 5\n\n[ Info: Saved animation to /tmp/jl_NAT2FUXkVe.gif\n\n\n\n\n\nThis has been an introduction to the applications of Turing and Flux in defining Bayesian neural networks.", "crumbs": [ "Documentation", "Using Turing - Tutorials", diff --git a/pr-previews/462/sitemap.xml b/pr-previews/462/sitemap.xml index 42de11ccc..2ea8fc406 100644 --- a/pr-previews/462/sitemap.xml +++ b/pr-previews/462/sitemap.xml @@ -2,126 +2,126 @@ https://turinglang.org/tutorials/02-logistic-regression/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/docs-14-using-turing-quick-start/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.169Z https://turinglang.org/tutorials/12-gplvm/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-11-using-turing-dynamichmc/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-09-using-turing-advanced/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-12-using-turing-guide/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.169Z https://turinglang.org/tutorials/15-gaussian-processes/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-05-for-developers-compiler/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-16-using-turing-external-samplers/index.html - 2024-05-29T12:19:11.819Z + 2024-05-29T18:24:19.169Z https://turinglang.org/tutorials/13-seasonal-time-series/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-10-using-turing-autodiff/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/11-probabilistic-pca/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/08-multinomial-logistic-regression/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/docs-00-getting-started/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-01-contributing-guide/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/00-introduction/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/09-variational-inference/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/05-linear-regression/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/03-bayesian-neural-network/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/docs-13-using-turing-performance-tips/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.169Z https://turinglang.org/tutorials/10-bayesian-differential-equations/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/07-poisson-regression/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/14-minituring/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/01-gaussian-mixture-model/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/04-hidden-markov-model/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/docs-06-for-developers-interface/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-08-using-turing/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-04-for-developers-abstractmcmc-turing/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z https://turinglang.org/tutorials/docs-15-using-turing-sampler-viz/index.html - 2024-05-29T12:19:11.819Z + 2024-05-29T18:24:19.169Z https://turinglang.org/tutorials/06-infinite-mixture-model/index.html - 2024-05-29T12:19:11.811Z + 2024-05-29T18:24:19.161Z https://turinglang.org/tutorials/docs-07-for-developers-variational-inference/index.html - 2024-05-29T12:19:11.815Z + 2024-05-29T18:24:19.165Z diff --git a/pr-previews/462/tutorials/03-bayesian-neural-network/index.html b/pr-previews/462/tutorials/03-bayesian-neural-network/index.html index f91218b4d..335ff1db8 100644 --- a/pr-previews/462/tutorials/03-bayesian-neural-network/index.html +++ b/pr-previews/462/tutorials/03-bayesian-neural-network/index.html @@ -489,7 +489,7 @@

Bayesian Neural Networks

using FillArrays using Lux using Plots -using ReverseDiff +using Tracker using Functors using LinearAlgebra @@ -538,129 +538,129 @@

Bayesian Neural Networks

- + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
@@ -876,17 +876,11 @@

Building a Neura 
# Perform inference.
-N = 500
-ch = sample(bayes_nn(reduce(hcat, xs), ts), NUTS(; adtype=AutoReverseDiff()), N);
+N = 2_000 +ch = sample(bayes_nn(reduce(hcat, xs), ts), NUTS(; adtype=AutoTracker()), N);
-
┌ Warning: Lux.apply(m::Lux.AbstractExplicitLayer, x::AbstractArray{<:ReverseDiff.TrackedReal}, ps, st) input was corrected to Lux.apply(m::Lux.AbstractExplicitLayer, x::ReverseDiff.TrackedArray}, ps, st).
-│ 
-│ 1. If this was not the desired behavior overload the dispatch on `m`.
-│ 
-│ 2. This might have performance implications. Check which layer was causing this problem using `Lux.Experimental.@debug_mode`.
-└ @ LuxReverseDiffExt ~/.julia/packages/Lux/PsbZF/ext/LuxReverseDiffExt.jl:25
-┌ Info: Found initial step size
-└   ϵ = 0.8
+
┌ Info: Found initial step size
+└   ϵ = 0.4

Now we extract the parameter samples from the sampled chain as θ (this is of size 5000 x 20 where 5000 is the number of iterations and 20 is the number of parameters). We’ll use these primarily to determine how good our model’s classifier is.

@@ -921,180 +915,180 @@

Prediction Visual - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + +

The contour plot above shows that the MAP method is not too bad at classifying our data.

@@ -1129,180 +1123,181 @@

Prediction Visual - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + +

Suppose we are interested in how the predictive power of our Bayesian neural network evolved between samples. In that case, the following graph displays an animation of the contour plot generated from the network weights in samples 1 to 1,000.

@@ -1316,10 +1311,10 @@

Prediction Visual contour!(x1_range, x2_range, Z; title="Iteration $i", clim=(0, 1)) end every 5
-
[ Info: Saved animation to /tmp/jl_56xULkjare.gif
+
[ Info: Saved animation to /tmp/jl_NAT2FUXkVe.gif
- +

This has been an introduction to the applications of Turing and Flux in defining Bayesian neural networks.