Manual section: | 3 |
---|
import std [from "path"] ;
STRING toupper(STRING s)
STRING tolower(STRING s)
VOID set_ip_tos(INT tos)
REAL random(REAL lo, REAL hi)
VOID log(STRING s)
VOID syslog(INT priority, STRING s)
STRING fileread(STRING)
BOOL file_exists(STRING path)
VOID collect(HEADER hdr, STRING sep)
DURATION duration(STRING s, DURATION fallback)
INT integer(STRING s, INT fallback)
IP ip(STRING s, IP fallback, BOOL resolve)
REAL real(STRING s, REAL fallback)
INT real2integer(REAL r, INT fallback)
TIME real2time(REAL r, TIME fallback)
INT time2integer(TIME t, INT fallback)
REAL time2real(TIME t, REAL fallback)
BOOL healthy(BACKEND be)
INT port(IP ip)
VOID rollback(HTTP h)
VOID timestamp(STRING s)
STRING querysort(STRING)
BOOL cache_req_body(BYTES size)
STRING strstr(STRING s1, STRING s2)
TIME time(STRING s, TIME fallback)
STRING getenv(STRING name)
VOID late_100_continue(BOOL late)
BOOL syntax(REAL)
BOOL fnmatch(STRING pattern, STRING subject, BOOL pathname, BOOL noescape, BOOL period)
vmod_std contains basic functions which are part and parcel of Varnish, but which for reasons of architecture fit better in a VMOD.
One particular class of functions in vmod_std is the conversions functions which all have the form:
TYPE type(STRING, TYPE)
These functions attempt to convert STRING to the TYPE, and if that fails, they return the second argument, which must have the given TYPE.
Logs the string s to syslog tagged with priority. priority is formed by ORing the facility and level values. See your system's syslog.h file for possible values.
Notice: Unlike VCL and other functions in the std vmod, this function will not fail VCL processing for workspace overflows: For an out of workspace condition, the syslog() function has no effect.
std.syslog(9, "Something is wrong");
This will send a message to syslog using LOG_USER | LOG_ALERT.
Consider that the entire contents of the file appear in the string that is returned, including newlines that may result in invalid headers if std.fileread() is used to form a header. In that case, you may need to modify the string, for example with regsub():
set beresp.http.served-by = regsub(std.fileread("/etc/hostname"), "\R$", "");
Collapses multiple hdr headers into one long header. The default separator sep is the standard comma separator to use when collapsing headers, with an additional whitespace for pretty printing.
Care should be taken when collapsing headers. In particular collapsing Set-Cookie will lead to unexpected results on the browser side.
Converts the string s to the first IP number returned by the system library function getaddrinfo(3). If conversion fails, fallback will be returned.
If resolve is false, getaddrinfo() is called using AI_NUMERICHOST to avoid network lookups. This makes "pure" IP strings cheaper to convert.
Caches the request body if it is smaller than size. Returns true if the body was cached, false otherwise.
Normally the request body is not available after sending it to the backend. By caching it is possible to retry pass operations, e.g. POST and PUT.
Returns a string beginning at the first occurrence of the string s2 in the string s1, or an empty string if s2 is not found.
Note that the comparison is case sensitive.
This will check if the content of req.http.restrict occurs anywhere in req.url.
Converts the string s to a time. If conversion fails, fallback will be returned.
Supported formats:
Return environment variable name or the empty string.
See getenv(3)
Controls when varnish reacts to an Expect: 100-continue client request header.
Varnish always generates a 100 Continue response if requested by the client trough the Expect: 100-continue header when waiting for request body data.
But, by default, the 100 Continue response is already generated immediately after vcl_recv returns to reduce latencies under the assumption that the request body will be read eventually.
Calling std.late_100_continue(true) in vcl_recv will cause the 100 Continue response to only be sent when needed. This may cause additional latencies for processing request bodies, but is the correct behavior by strict interpretation of RFC7231.
This function has no effect outside vcl_recv and after calling std.cache_req_body() or any other function consuming the request body.
BOOL fnmatch(
STRING pattern,
STRING subject,
BOOL pathname=1,
BOOL noescape=0,
BOOL period=0
)
Shell-style pattern matching; returns true if subject matches pattern, where pattern may contain wildcard characters such as * or ?.
The match is executed by the implementation of fnmatch(3) on your system. The rules for pattern matching on most systems include the following:
If pathname is true, then the forward slash character / is only matched literally, and never matches *, ? or a bracket expression. Otherwise, / may match one of those patterns. By default, pathname is true.
If noescape is true, then the backslash character \ is matched as an ordinary character. Otherwise, \ is an escape character, and matches the character that follows it in the pattern. For example, \\ matches \ when noescape is true, and \\ when false. By default, noescape is false.
If period is true, then a leading period character . only matches literally, and never matches *, ? or a bracket expression. A period is leading if it is the first character in subject; if pathname is also true, then a period that immediately follows a / is also leading (as in "/."). By default, period is false.
fnmatch() invokes VCL failure and returns false if either of pattern or subject is NULL -- for example, if an unset header is specified.
Copyright (c) 2010-2017 Varnish Software AS
All rights reserved.
Author: Poul-Henning Kamp <phk@FreeBSD.org>
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
Manual section: | 3 |
---|
import directors [from "path"] ;
new xround_robin = round_robin()
VOID xround_robin.add_backend(BACKEND)
VOID xround_robin.remove_backend(BACKEND)
BACKEND xround_robin.backend()
new xfallback = fallback(BOOL sticky)
VOID xfallback.add_backend(BACKEND)
VOID xfallback.remove_backend(BACKEND)
BACKEND xfallback.backend()
new xrandom = random()
VOID xrandom.add_backend(BACKEND, REAL)
VOID xrandom.remove_backend(BACKEND)
BACKEND xrandom.backend()
new xhash = hash()
VOID xhash.add_backend(BACKEND, REAL)
VOID xhash.remove_backend(BACKEND)
BACKEND xhash.backend(STRING)
new xshard = shard()
VOID xshard.set_warmup(REAL probability)
VOID xshard.set_rampup(DURATION duration)
VOID xshard.associate(BLOB param)
BOOL xshard.add_backend(BACKEND backend, [STRING ident], [DURATION rampup])
BOOL xshard.remove_backend([BACKEND backend], [STRING ident])
BOOL xshard.clear()
BOOL xshard.reconfigure(INT replicas)
INT xshard.key(STRING)
BACKEND xshard.backend([ENUM by], [INT key], [BLOB key_blob], [INT alt], [REAL warmup], [BOOL rampup], [ENUM healthy], [BLOB param], [ENUM resolve])
VOID xshard.debug(INT)
new xshard_param = shard_param()
VOID xshard_param.clear()
VOID xshard_param.set([ENUM by], [INT key], [BLOB key_blob], [INT alt], [REAL warmup], [BOOL rampup], [ENUM healthy])
STRING xshard_param.get_by()
INT xshard_param.get_key()
INT xshard_param.get_alt()
REAL xshard_param.get_warmup()
BOOL xshard_param.get_rampup()
STRING xshard_param.get_healthy()
BLOB xshard_param.use()
vmod_directors enables backend load balancing in Varnish.
The module implements load balancing techniques, and also serves as an example on how one could extend the load balancing capabilities of Varnish.
To enable load balancing you must import this vmod (directors).
Then you define your backends. Once you have the backends declared you can add them to a director. This happens in executed VCL code. If you want to emulate the previous behavior of Varnish 3.0 you can just initialize the directors in vcl_init, like this:
sub vcl_init {
new vdir = directors.round_robin();
vdir.add_backend(backend1);
vdir.add_backend(backend2);
}
As you can see there is nothing keeping you from manipulating the directors elsewhere in VCL. So, you could have VCL code that would add more backends to a director when a certain URL is called.
Note that directors can use other directors as backends.
Create a round robin director.
This director will pick backends in a round robin fashion.
Create a fallback director.
A fallback director will try each of the added backends in turn, and return the first one that is healthy.
If sticky is set to true, the director will keep using the healthy backend, even if a higher-priority backend becomes available. Once the whole backend list is exhausted, it'll start over at the beginning.
Add a backend to the director.
Note that the order in which this is done matters for the fallback director.
Create a random backend director.
The random director distributes load over the backends using a weighted random probability distribution. The "testable" random generator in varnishd is used, which enables deterministic tests to be run (See: d00004.vtc).
Add a backend to the director with a given weight.
Each backend will receive approximately 100 * (weight / (sum(all_added_weights))) per cent of the traffic sent to this director.
Create a hashing backend director.
The director chooses the backend server by computing a hash/digest of the string given to .backend().
Commonly used with client.ip or a session cookie to get sticky sessions.
Add a backend to the director with a certain weight.
Weight is used as in the random director. Recommended value is 1.0 unless you have special needs.
Pick a backend from the backend director.
Use the string or list of strings provided to pick the backend.
Create a shard director.
Note that the shard director needs to be configured using at least one shard.add_backend() call(s) followed by a shard.reconfigure() call before it can hand out backends.
_Note_ that due to various restrictions (documented below), it is recommended to use the shard director on the backend side.
The shard director selects backends by a key, which can be provided directly or derived from strings. For the same key, the shard director will always return the same backend, unless the backend configuration or health state changes. Conversely, for differing keys, the shard director will likely choose different backends. In the default configuration, unhealthy backends are not selected.
The shard director resembles the hash director, but its main advantage is that, when the backend configuration or health states change, the association of keys to backends remains as stable as possible.
In addition, the rampup and warmup features can help to further improve user-perceived response times.
This basic technique allows for numerous applications like optimizing backend server cache efficiency, Varnish clustering or persisting sessions to servers without keeping any state, and, in particular, without the need to synchronize state between nodes of a cluster of Varnish servers:
Many applications use caches for data objects, so, in a cluster of application servers, requesting similar objects from the same server may help to optimize efficiency of such caches.
For example, sharding by URL or some id component of the url has been shown to drastically improve the efficiency of many content management systems.
As special case of the previous example, in clusters of Varnish servers without additional request distribution logic, each cache will need store all hot objects, so the effective cache size is approximately the smallest cache size of any server in the cluster.
Sharding allows to segregate objects within the cluster such that each object is only cached on one of the servers (or on one primary and one backup, on a primary for long and others for short etc...). Effectively, this will lead to a cache size in the order of the sum of all individual caches, with the potential to drastically increase efficiency (scales by the number of servers).
Another application is to implement persistence of backend requests, such that all requests sharing a certain criterion (such as an IP address or session ID) get forwarded to the same backend server.
When used with clusters of varnish servers, the shard director will, if otherwise configured equally, make the same decision on all servers. In other words, requests sharing a common criterion used as the shard key will be balanced onto the same backend server(s) no matter which Varnish server handles the request.
The drawbacks are:
When .reconfigure() is called, a consistent hashing circular data structure gets built from the last 32 bits of SHA256 hash values of <ident><n> (default ident being the backend name) for each backend and for a running number n from 1 to replicas. Hashing creates the seemingly random order for placement of backends on the consistent hashing ring.
When .backend() is called, a load balancing key gets generated unless provided. The smallest hash value in the circle is looked up that is larger than the key (searching clockwise and wrapping around as necessary). The backend for this hash value is the preferred backend for the given key.
If a healthy backend is requested, the search is continued linearly on the ring as long as backends found are unhealthy or all backends have been checked. The order of these "alternative backends" on the ring is likely to differ for different keys. Alternative backends can also be selected explicitly.
On consistent hashing see:
Failing methods should report errors to VSL with the Error tag, so when configuring the shard director, you are advised to check:
varnishlog -I Error:^shard
Set the default warmup probability. See the warmup parameter of shard.backend(). If probability is 0.0 (default), warmup is disabled.
Set the default rampup duration. See rampup parameter of shard.backend(). If duration is 0 (default), rampup is disabled.
Associate a default obj_shard_param object or clear an association.
The value of the param argument must be a call to the func_shard_param.use method. No argument clears the association.
The association can be changed per backend request using the param argument of func_shard.backend.
BOOL xshard.add_backend(
BACKEND backend,
[STRING ident],
[DURATION rampup]
)
Add a backend backend to the director.
ident: Optionally specify an identification string for this backend, which will be hashed by shard.reconfigure() to construct the consistent hashing ring. The identification string defaults to the backend name.
ident allows to add multiple instances of the same backend.
rampup: Optionally specify a specific rampup time for this backend. Otherwise, the per-director rampup time is used (see VOID xshard.set_rampup(DURATION duration=0)).
NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.
BOOL xshard.remove_backend(
[BACKEND backend=0],
[STRING ident=0]
)
Remove backend(s) from the director. Either backend or ident must be specified. ident removes a specific instance. If backend is given without ident, all instances of this backend are removed.
NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.
Remove all backends from the director.
NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.
Reconfigure the consistent hashing ring to reflect backend changes.
This method must be called at least once before the director can be used.
Convenience method to generate a sharding key for use with the key argument to the shard.backend() method by hashing the given string with SHA256.
To generate sharding keys using other hashes, use a custom vmod like vmod blobdigest with the key_blob argument of the shard.backend() method.
BACKEND xshard.backend(
[ENUM {HASH, URL, KEY, BLOB} by=HASH],
[INT key],
[BLOB key_blob],
[INT alt=0],
[REAL warmup=-1],
[BOOL rampup=1],
[ENUM {CHOSEN, IGNORE, ALL} healthy=CHOSEN],
[BLOB param],
[ENUM {NOW, LAZY} resolve]
)
Lookup a backend on the consistent hashing ring.
This documentation uses the notion of an order of backends for a particular shard key. This order is deterministic but seemingly random as determined by the consistent hashing algorithm and is likely to differ for different keys, depending on the number of backends and the number of replicas. In particular, the backend order referred to here is _not_ the order given when backends are added.
by how to determine the sharding key
key lookup key with by=KEY
the shard.key() function may come handy to generate a sharding key from custom strings.
key_blob lookup key with by=BLOB
Currently, this uses the first 4 bytes from the given blob in network byte order (big endian), left-padded with zeros for blobs smaller than 4 bytes.
alt alternative backend selection
Select the alt-th alternative backend for the given key.
This is particularly useful for retries / restarts due to backend errors: By setting alt=req.restarts or alt=bereq.retries with healthy=ALL, another server gets selected.
The rampup and warmup features are only active for alt==0
rampup slow start for servers which just went healthy
If alt==0 and the chosen backend is in its rampup period, with a probability proportional to the fraction of time since the backup became healthy to the rampup period, return the next alternative backend, unless this is also in its rampup period.
The default rampup interval can be set per shard director using the set_rampup() method or specifically per backend with the set_backend() method.
warmup probabilistic alternative server selection
possible values: -1, 0..1
-1: use the warmup probability from the director definition
Only used for alt==0: Sets the ratio of requests (0.0 to 1.0) that goes to the next alternate backend to warm it up when the preferred backend is healthy. Not active if any of the preferred or alternative backend are in rampup.
warmup=0.5 is a convenient way to spread the load for each key over two backends under normal operating conditions.
healthy
CHOSEN: Return a healthy backend if possible.
For alt==0, return the first healthy backend or none.
For alt > 0, ignore the health state of backends skipped for alternative backend selection, then return the next healthy backend. If this does not exist, return the last healthy backend of those skipped or none.
IGNORE: Completely ignore backend health state
Just return the first or alt-th alternative backend, ignoring health state. Ignore rampup and warmup.
ALL: Check health state also for alternative backend selection
For alt > 0, return the alt-th alternative backend of all those healthy, the last healthy backend found or none.
resolve
default: LAZY in vcl_init{}, NOW otherwise
NOW: look up a backend and return it.
Can not be used in vcl_init{}.
LAZY: return an instance of this director for later backend resolution.
LAZY mode is required for referencing shard director instances, for example as backends for other directors (director layering).
In vcl_init{} and on the client side, LAZY mode can not be used with any other argument.
On the backend side, parameters from arguments or an associated parameter set affect the shard director instance for the backend request irrespective of where it is referenced.
param
Use or associate a parameter set. The value of the param argument must be a call to the func_shard_param.use method.
default: as set by func_shard.associate or unset.
for resolve=NOW take parameter defaults from the obj_shard_param parameter set
for resolve=LAZY associate the obj_shard_param parameter set for this backend request
Implementation notes for use of parameter sets with resolve=LAZY:
intentionally undocumented
Create a shard parameter set.
A parameter set allows for re-use of func_shard.backend arguments across many shard director instances and simplifies advanced use cases (e.g. shard director with custom parameters layered below other directors).
Parameter sets have two scopes:
The per-VCL scope defines defaults for the per backend scope. Any changes to a parameter set in backend context only affect the respective backend request.
Parameter sets can not be used in client context.
Reset the parameter set to default values as documented for func_shard.backend.
This method may not be used in client context
VOID xshard_param.set(
[ENUM {HASH, URL, KEY, BLOB} by],
[INT key],
[BLOB key_blob],
[INT alt],
[REAL warmup],
[BOOL rampup],
[ENUM {CHOSEN, IGNORE, ALL} healthy]
)
Change the given parameters of a parameter set as documented for func_shard.backend.
This method may not be used in client context
Get a string representation of the by enum argument which denotes how a shard director using this parameter object would derive the shard key. See func_shard.backend.
Get the key which a shard director using this parameter object would use. See func_shard.backend.
Get the alt parameter which a shard director using this parameter object would use. See func_shard.backend.
Get the warmup parameter which a shard director using this parameter object would use. See func_shard.backend.
Get the rampup parameter which a shard director using this parameter object would use. See func_shard.backend.
Get a string representation of the healthy enum argument which a shard director using this parameter object would use. See func_shard.backend.
This method may only be used in backend context.
For use with the param argument of func_shard.backend to associate this shard parameter set with a shard director.
Development of a previous version of the shard director was partly sponsored by Deutsche Telekom AG - Products & Innovation.
Development of a previous version of the shard director was partly sponsored by BILD GmbH & Co KG.
This document is licensed under the same licence as Varnish
itself. See LICENCE for details.
Copyright (c) 2013-2015 Varnish Software AS
Copyright 2009-2018 UPLEX - Nils Goroll Systemoptimierung
All rights reserved.
Authors: Poul-Henning Kamp <phk@FreeBSD.org>
Julian Wiesener <jw@uplex.de>
Nils Goroll <slink@uplex.de>
Geoffrey Simmons <geoff@uplex.de>
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
Manual section: | 3 |
---|
import vtc [from "path"] ;
VOID barrier_sync(STRING addr, DURATION timeout)
BACKEND no_backend()
STEVEDORE no_stevedore()
VOID panic(STRING)
VOID sleep(DURATION)
VOID workspace_alloc(ENUM, INT size)
INT workspace_free(ENUM)
VOID workspace_snapshot(ENUM)
VOID workspace_reset(ENUM)
BOOL workspace_overflowed(ENUM)
VOID workspace_overflow(ENUM)
INT typesize(STRING)
The goal for this VMOD is to provide VCL users and VMOD authors means to test corner cases or reach certain conditions with varnishtest.
When writing test cases, the most common pattern is to start a mock server instance, a Varnish instance, and spin up a mock client. Those entities run asynchronously, and others exist like background processes (process) or log readers (logexpect). While you can synchronize with individual entities and wait for their completion, you must use a barrier if you need to synchronize two or more entities, or wait until a certain point instead of completion.
Not only is it possible to synchronize between test entities, with the barrier_sync function you can even synchronize VCL code:
sub vcl_recv {
# wait for some barrier b1 to complete
vtc.barrier_sync("${b1_sock}");
}
If the function fails to synchronize with the barrier for some reason, or if it reaches the optional timeout, it fails the VCL transaction.
Fails at backend selection.
Fails at storage selection.
It can be useful to crash the child process in order to test the robustness of a VMOD.
Block the current worker thread.
It can be useful to put a workspace in a given state when testing corner cases like resource exhaustion for a transaction, especially for VMOD development. All functions available allow to pick which workspace you need to tamper with, available values are client, backend, session and thread.
VOID workspace_alloc(
ENUM {client, backend, session, thread},
INT size
)
Allocate and zero out memory from a workspace. A negative size will allocate as much as needed to leave that many bytes free. The actual allocation size may be higher to comply with memory alignment requirements of the CPU architecture. A failed allocation fails the transaction.
Find how much unallocated space there is left in a workspace.
VOID workspace_snapshot(ENUM {client, backend, session, thread})
Snapshot a workspace. Only one snapshot may be active at a time.
VOID workspace_reset(ENUM {client, backend, session, thread})
Reset to the previous snapshot of a workspace, it must be the same workspace too.
BOOL workspace_overflowed(ENUM {client, backend, session, thread})
Find whether the workspace overflow mark is set or not.
VOID workspace_overflow(ENUM {client, backend, session, thread})
Mark a workspace as overflowed.
Returns the size in bytes of a collection of C-datatypes:
This can be useful for VMOD authors in conjunction with workspace operations.
Copyright (c) 2017 Varnish Software AS
All rights reserved.
Author: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
Manual section: | 3 |
---|
import purge [from "path"] ;
INT hard()
INT soft(DURATION ttl, DURATION grace, DURATION keep)
vmod_purge contains functions that offer a finer-grained control than the purge transition in vcl_recv. The functions can only be called from vcl_hit or vcl_miss and they should in general be used in both to ensure that all variants of a same object are taken care of.
sub vcl_recv {
if (req.method == "PURGE") {
if (client.ip !~ purge_acl) {
return (synth(405));
}
return (hash);
}
}
sub my_purge {
set req.http.purged = purge.hard();
if (req.http.purged == "0") {
return (synth(404));
}
else {
return (synth(200));
}
}
sub vcl_hit {
if (req.method == "PURGE") {
call my_purge;
}
}
sub vcl_miss {
if (req.method == "PURGE") {
call my_purge;
}
}
sub vcl_synth {
if (req.method == "PURGE") {
if (req.http.purged) {
set resp.http.purged = req.http.purged;
}
return (deliver);
}
}
INT soft(DURATION ttl=0, DURATION grace=-1, DURATION keep=-1)
Copyright (c) 2017 Varnish Software AS
All rights reserved.
Author: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
Manual section: | 3 |
---|
import blob [from "path"] ;
BLOB decode(ENUM decoding, INT length, STRING encoded)
STRING encode(ENUM encoding, ENUM case, BLOB blob)
STRING transcode(ENUM decoding, ENUM encoding, ENUM case, INT length, STRING encoded)
BOOL same(BLOB, BLOB)
BOOL equal(BLOB, BLOB)
INT length(BLOB)
BLOB sub(BLOB, BYTES length, BYTES offset)
new xblob = blob(ENUM decoding, STRING encoded)
BLOB xblob.get()
STRING xblob.encode(ENUM encoding, ENUM case)
This VMOD provides utility functions and an object for the VCL data type BLOB, which may contain arbitrary data of any length.
Examples:
sub vcl_init {
# Create blob objects from encodings such as base64 or hex.
new myblob = blob.blob(BASE64, "Zm9vYmFy");
new yourblob = blob.blob(encoded="666F6F", decoding=HEX);
}
sub vcl_deliver {
# The .get() method retrieves the BLOB from an object.
set resp.http.MyBlob-As-Hex
= blob.encode(blob=myblob.get(), encoding=HEX);
# The .encode() method efficiently retrieves an encoding.
set resp.http.YourBlob-As-Base64 = yourblob.encode(BASE64);
# decode() and encode() functions convert blobs to text and
# vice versa at runtime.
set resp.http.Base64-Encoded
= blob.encode(BASE64,
blob=blob.decode(HEX,
encoded=req.http.Hex-Encoded));
}
sub vcl_recv {
# transcode() converts from one encoding to another.
# case=UPPER specifies upper-case hex digits A-F.
set req.http.Hex-Encoded
= blob.transcode(decoding=BASE64, encoding=HEX,
case=UPPER, encoded="YmF6");
# transcode() from URL to IDENTITY effects a URL decode.
set req.url = blob.transcode(encoded=req.url, decoding=URL);
# transcode() from IDENTITY to URL effects a URL encode.
set req.http.url_urlcoded
= blob.transcode(encoded=req.url, encoding=URL);
}
Binary-to-text encoding schemes are specified by ENUMs in the VMOD's constructor, methods and functions. Decodings convert a (possibly concatenated) string into a blob, while encodings convert a blob into a string.
ENUM values for an encoding scheme can be one of:
Empty strings are decoded into a "null blob" (of length 0), and conversely a null blob is encoded as the empty string.
For encodings with HEX or URL, you may also specify a case ENUM with one of the values LOWER, UPPER or DEFAULT to produce a string with lower- or uppercase hex digits (in [a-f] or [A-F]). The default value for case is DEFAULT, which for HEX and URL means the same as LOWER.
The case ENUM is not relevant for decodings; HEX or URL strings to be decoded as BLOBs may have hex digits in either case, or in mixed case.
The case ENUM MUST be set to DEFAULT for the other encodings (BASE64* and IDENTITY). You cannot, for example, produce an uppercase string by using the IDENTITY scheme with case=UPPER. To change the case of a string, use the toupper or tolower functions from vmod_std.
The simplest encoding converts between the BLOB and STRING data types, leaving the contents byte-identical.
Note that a BLOB may contain a null byte at any position before its end; if such a BLOB is decoded with IDENTITY, the resulting STRING will have a null byte at that position. Since VCL strings, like C strings, are represented with a terminating null byte, the string will be truncated, appearing to contain less data than the original blob. For example:
# Decode from the hex encoding for "foo\0bar".
# The header will be seen as "foo".
set resp.http.Trunced-Foo1
= blob.encode(IDENTITY, blob=blob.decode(HEX,
encoded="666f6f00626172"));
IDENTITY is the default encoding and decoding. So the above can also be written as:
# Decode from the hex encoding for "foo\0bar".
# The header will be seen as "foo".
set resp.http.Trunced-Foo2
= blob.encode(blob=blob.decode(HEX, encoded="666f6f00626172"));
The case ENUM MUST be set to DEFAULT for IDENTITY encodings.
The base64 encoding schemes use 4 characters to encode 3 bytes. There are no newlines or maximal line lengths -- whitespace is not permitted.
The BASE64 encoding uses the alphanumeric characters, + and /; and encoded strings are padded with the = character so that their length is always a multiple of four.
The BASE64URL encoding also uses the alphanumeric characters, but - and _ instead of + and /, so that an encoded string can be used safely in a URL. This scheme also uses the padding character =.
The BASE64URLNOPAD encoding uses the same alphabet as BASE6URL, but leaves out the padding. Thus the length of an encoding with this scheme is not necessarily a multiple of four.
The case ENUM MUST be set to DEFAULT for for all of the BASE64* encodings.
The HEX encoding scheme converts hex strings into blobs and vice versa. For encodings, you may use the case ENUM to specify upper- or lowercase hex digits A through f (default DEFAULT, which means the same as LOWER). A prefix such as 0x is not used for an encoding and is illegal for a decoding.
If a hex string to be decoded has an odd number of digits, it is decoded as if a 0 is prepended to it; that is, the first digit is interpreted as representing the least significant nibble of the first byte. For example:
# The concatenated string is "abcdef0", and is decoded as "0abcdef0".
set resp.http.First = "abc";
set resp.http.Second = "def0";
set resp.http.Hex-Decoded
= blob.encode(HEX, blob=blob.decode(HEX,
encoded=resp.http.First + resp.http.Second));
The URL decoding replaces any %<2-hex-digits> substrings with the binary value of the hexadecimal number after the % sign.
The URL encoding implements "percent encoding" as per RFC3986. The case ENUM determines the case of the hex digits, but does not affect alphabetic characters that are not percent-encoded.
BLOB decode(
ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
INT length=0,
STRING encoded
)
Returns the BLOB derived from the string encoded according to the scheme specified by decoding.
If length > 0, only decode the first length characters of the encoded string. If length <= 0 or greater than the length of the string, then decode the entire string. The default value of length is 0.
decoding defaults to IDENTITY.
Example:
blob.decode(BASE64, encoded="Zm9vYmFyYmF6");
# same with named parameters
blob.decode(encoded="Zm9vYmFyYmF6", decoding=BASE64);
# convert string to blob
blob.decode(encoded="foo");
STRING encode(
ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
BLOB blob
)
Returns a string representation of the BLOB blob as specified by encoding. case determines the case of hex digits for the HEX and URL encodings, and is ignored for the other encodings.
encoding defaults to IDENTITY, and case defaults to DEFAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings, and is the required value for the other encodings.
Example:
set resp.http.encode1
= blob.encode(HEX,
blob=blob.decode(BASE64, encoded="Zm9vYmFyYmF6"));
# same with named parameters
set resp.http.encode2
= blob.encode(blob=blob.decode(encoded="Zm9vYmFyYmF6",
decoding=BASE64),
encoding=HEX);
# convert blob to string
set resp.http.encode3
= blob.encode(blob=blob.decode(encoded="foo"));
STRING transcode(
ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
INT length=0,
STRING encoded
)
Translates from one encoding to another, by first decoding the string encoded according to the scheme decoding, and then returning the encoding of the resulting blob according to the scheme encoding. case determines the case of hex digits for the HEX and URL encodings, and is ignored for other encodings.
As with decode(): If length > 0, only decode the first length characters of the encoded string, otherwise decode the entire string. The default value of length is 0.
decoding and encoding default to IDENTITY, and case defaults to DEFAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings, and is the required value for the other encodings.
Example:
set resp.http.Hex2Base64-1
= blob.transcode(HEX, BASE64, encoded="666f6f");
# same with named parameters
set resp.http.Hex2Base64-2
= blob.transcode(encoded="666f6f",
encoding=BASE64, decoding=HEX);
# URL decode -- recall that IDENTITY is the default encoding.
set resp.http.urldecoded
= blob.transcode(encoded="foo%20bar", decoding=URL);
# URL encode
set resp.http.urlencoded
= blob.transcode(encoded="foo bar", encoding=URL);
Returns true if and only if the two BLOB arguments are the same object, i.e. they specify exactly the same region of memory, or both are empty.
If the BLOBs are both empty (length is 0 and/or the internal pointer is NULL), then same() returns true. If any non-empty BLOB is compared to an empty BLOB, then same() returns false.
Returns true if and only if the two BLOB arguments have equal contents (possibly in different memory regions).
As with same(): If the BLOBs are both empty, then equal() returns true. If any non-empty BLOB is compared to an empty BLOB, then equal() returns false.
Returns the length of the BLOB.
Returns a new BLOB formed from length bytes of the BLOB argument starting at offset bytes from the start of its memory region. The default value of offset is 0B.
sub() fails and returns NULL if the BLOB argument is empty, or if offset + length requires more bytes than are available in the BLOB.
new xblob = blob(
ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
STRING encoded
)
Creates an object that contains the BLOB derived from the string encoded according to the scheme decoding.
Example:
new theblob1 = blob.blob(BASE64, encoded="YmxvYg==");
# same with named arguments
new theblob2 = blob.blob(encoded="YmxvYg==", decoding=BASE64);
# string as a blob
new stringblob = blob.blob(encoded="bazz");
Returns the BLOB created by the constructor.
Example:
set resp.http.The-Blob1 =
blob.encode(blob=theblob1.get());
set resp.http.The-Blob2 =
blob.encode(blob=theblob2.get());
set resp.http.The-Stringblob =
blob.encode(blob=stringblob.get());
STRING xblob.encode(
ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT
)
Returns an encoding of BLOB created by the constructor, according to the scheme encoding. case determines the case of hex digits for the HEX and URL encodings, and MUST be set to DEFAULT for the other encodings.
Example:
# blob as text
set resp.http.The-Blob = theblob1.encode();
# blob as base64
set resp.http.The-Blob-b64 = theblob1.encode(BASE64);
For any blob object, encoding ENC and case CASE, encodings via the .encode() method and the encode() function are equal:
# Always true:
blob.encode(ENC, CASE, blob.get()) == blob.encode(ENC, CASE)
But the object method is more efficient -- the encoding is computed once and cached (with allocation in heap memory), and the cached encoding is retrieved on every subsequent call. The encode() function computes the encoding on every call, allocating space for the string in Varnish workspaces.
So if the data in a BLOB are fixed at VCL initialization time, so that its encodings will always be the same, it is better to create a blob object. The VMOD's functions should be used for data that are not known until runtime.
The encoders, decoders and sub() may fail if there is insufficient space to create the new blob or string. Decoders may also fail if the encoded string is an illegal format for the decoding scheme. Encoders will fail for the IDENTITY and BASE64* encoding schemes if the case ENUM is not set to DEFAULT.
If any of the VMOD's methods, functions or constructor fail, then VCL failure is invoked, just as if return(fail) had been called in the VCL source. This means that:
The VMOD allocates memory in various ways for new blobs and strings. The blob object and its methods allocate memory from the heap, and hence they are only limited by available virtual memory.
The encode(), decode() and transcode() functions allocate Varnish workspace, as does sub() for the newly created BLOB. If these functions are failing, as indicated by "out of space" messages in the Varnish log (with the VCL_Error tag), then you will need to increase the varnishd parameters workspace_client and/or workspace_backend.
The transcode() function also allocates space on the stack for a temporary BLOB. If this function causes stack overflow, you may need to increase the varnishd parameter thread_pool_stack.
This document is licensed under the same conditions as Varnish itself.
See LICENSE for details.
Authors: Nils Goroll <nils.goroll@uplex.de>
Geoffrey Simmons <geoffrey.simmons@uplex.de>
Manual section: | 3 |
---|
import unix [from "path"] ;
STRING user()
STRING group()
INT uid()
INT gid()
This VMOD provides information about the credentials of the peer process (user and group of the process owner) that is connected to a Varnish listener via a Unix domain socket, if the platform supports it.
Examples:
import unix;
sub vcl_recv {
# Return "403 Forbidden" if the connected peer is
# not running as the user "trusteduser".
if (unix.user() != "trusteduser") {
return( synth(403) );
}
# Require the connected peer to run in the group
# "trustedgroup".
if (unix.group() != "trustedgroup") {
return( synth(403) );
}
# Require the connected peer to run under a specific numeric
# user id.
if (unix.uid() != 4711) {
return( synth(403) );
}
# Require the connected peer to run under a numeric group id.
if (unix.gid() != 815) {
return( synth(403) );
}
}
Obtaining the peer credentials is possible on a platform that supports one of the following:
On SunOS and friends, the PRIV_PROC_INFO privilege set is added to the Varnish child process while the VMOD is loaded, see setppriv(2).
On most platforms, the value returned is the effective user or group that was valid when the peer process initiated the connection.
Return the user name of the peer process owner.
Return the group name of the peer process owner.
Return the numeric user id of the peer process owner.
Return the numeric group id of the peer process owner.
All functions in this VMOD are subject to the following constraints:
None of them may be called in vcl_init or vcl_fini. If one of them is called in vcl_init, then the VCL program will fail to load, with an error message from the VMOD.
If called on a platform that is not supported, then VCL failure is invoked. An error message is written to the log (with the VCL_Error tag), and for all VCL subroutines except for vcl_synth, control is directed immediately to vcl_synth, with the response status set to 503 and the reason string set to "VCL failed".
If the failure occurs during vcl_synth, then vcl_synth is aborted, and the the response line "503 VCL failed" is sent.
If the current listener is not a Unix domain socket, or if the attempt to read credentials fails, then a VCL_Error message is written to the log. The STRING functions (vmod_user and vmod_group) return NULL, while the INT functions (vmod_uid and vmod_gid) return -1.
This document is licensed under the same conditions as Varnish itself.
See LICENSE for details.
Authors: Geoffrey Simmons <geoffrey.simmons@uplex.de>
Manual section: | 3 |
---|
import proxy [from "path"] ;
STRING alpn()
STRING authority()
BOOL is_ssl()
BOOL client_has_cert_sess()
BOOL client_has_cert_conn()
INT ssl_verify_result()
STRING ssl_version()
STRING client_cert_cn()
STRING ssl_cipher()
STRING cert_sign()
STRING cert_key()
vmod_proxy contains functions to extract proxy-protocol-v2 TLV attributes as described in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.
Copyright (c) 2018 GANDI SAS
All rights reserved.
Author: Emmanuel Hocdet <manu@gandi.net>
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.