Trims, sanitizes & scrubs data based on struct tags (go, golang)

Conform- keep user input in check (go, golang)

Trim, sanitize, and modify struct string fields in place, based on tags.

Update Jan 12, 2016 -- Now also works with embedded structs

Turns this...

type Person struct {
	FirstName string `conform:"name"`
	LastName  string `conform:"ucfirst,trim"`
	Email     string `conform:"email"`
	CamelCase string `conform:"camel"`
	UserName  string `conform:"snake"`
	Slug      string `conform:"slug"`
	Blurb     string `conform:"title"`
	Left      string `conform:"ltrim"`
	Right     string `conform:"rtrim"`
}

p1 := Person{
	" LEE ",
	"     Benson",
	"   [email protected]  ",
	"I love new york city",
	"lee benson",
	"LeeBensonWasHere",
	"this is a little bit about me...",
	"    Left trim   ",
	"    Right trim  ",
}

Into this...

p2 := p1 // <-- copy the Person struct into a new one, to see the difference
conform.Strings(&p2) // <-- this does the work

/*
	p1 (left) vs. p2 (right)

	FirstName: ' LEE ' -> 'Lee'
	LastName: '     Benson' -> 'Benson'
	Email: '   [email protected]  ' -> '[email protected]'
	CamelCase: 'I love new york city' -> 'ILoveNewYorkCity'
	UserName: 'lee benson' -> 'lee_benson'
	Slug: 'LeeBensonWasHere' -> 'lee-benson-was-here'
	Blurb: 'this is a little bit about me...' -> 'This Is A Little Bit About Me...'
	Left: '    Left trim   ' -> 'Left trim   '
	Right: '    Right trim  ' -> '    Right trim'
*/

Why?

Conform helps you fix and format user strings quickly, without writing functions.

If you do form processing with Gorilla Schema or similar, you probably shuttle user data into structs using tags. Adding a conform tag to your string field gives you "first pass" clean up against user input.

Use it for names, e-mail addresses, URL slugs, or any other form field where formatting matters.

Conform doesn't attempt any kind of validation on your fields. Check out govalidator for a slew of common validation funcs, or validator which is an uber-flexible Swiss Army knife for validating pretty much any kind of data you can imagine. Both have struct tag syntax and can be used with conform.

How to use

Grab the package from the command line with:

go get github.com/leebenson/conform

And import in the usual way in your Go app:

import "github.com/leebenson/conform"

Add a conform tag to your structs, for all of the string fields that you want Conform to transform. Add the name of the transform (known as the "tag") in double quotes, and separate multiple tags with commas. Example: conform:"trim,lowercase"

To format in place, pass your struct pointer to conform.Strings.

Note: your struct will be edited in place. This will OVERWRITE any data that is already stored in your string fields.

Here's an example that formats e-mail addresses:

package main

import (
		"fmt"
		"github.com/leebenson/conform"
)

type UserForm struct {
	Email string `conform:"email"`
}

func main() {
	input := UserForm{
		Email: "   [email protected]  ",
	}
	conform.Strings(&input) // <-- pass in a pointer to your struct
	fmt.Println(input.Email) // prints "[email protected]"
}

Using with Gorilla Schema

Just add a conform tag along with your Gorilla schema tags:

// ...

import (
	"net/http"

	"github.com/gorilla/schema"
	"github.com/leebenson/conform"
)

// the struct that will be filled from the post request...
type newUserForm struct {
	FirstName string    `schema:"firstName" conform:"name"`
	Email     string    `schema:"emailAddress" conform:"email"`
	Password  string    `schema:"password"`    // <-- no tag? no change
	Dob       time.Time `schema:"dateOfBirth"` // <-- non-strings ignored by conform
}

// ProcessNewUser attempts to register a new user
func ProcessNewUser(r *http.Request) error {
	form := new(newUserForm)
	schema.NewDecoder().Decode(form, r.PostForm) // <-- Gorilla Schema
	conform.Strings(form)                       // <-- Conform.  Pass in the same pointer that Schema used
	// ...
}

// HTTP handlers, etc...

Godoc

See the public API / exported methods on Godoc.

Tags

You can use multiple tags in the format of conform:"tag1,tag2"

trim


Trims leading and trailing spaces. Example: " string " -> "string"

ltrim


Trims leading spaces only. Example: " string " -> "string "

rtrim


Trims trailing spaces only. Example: " string " -> " string"

lower


Converts string to lowercase. Example: "STRING" -> "string"

upper


Converts string to uppercase. Example: "string" -> "STRING"

title


Converts string to Title Case, e.g. "this is a sentence" -> "This Is A Sentence"

camel


Converts to camel case via stringUp, Example provided by library: this is it => thisIsIt, this\_is\_it => thisIsIt, this-is-it => thisIsIt

snake


Converts to snake_case. Example: "CamelCase" -> "camel_case", "regular string" -> "regular_string" Special thanks to snaker for inspiration (credited in license)

slug


Turns strings into slugs. Example: "CamelCase" -> "camel-case", "blog title here" -> "blog-title-here"

ucfirst


Uppercases first character. Example: "all lower" -> "All lower"

name


Trims, strips numbers and special characters (except dashes and spaces separating names), converts multiple spaces and dashes to single characters, title cases multiple names. Example: "3493€848Jo-s$%£@Ann " -> "Jo-Ann", " ~~ The Dude ~~" -> "The Dude", "**susan**" -> "Susan", " hugh fearnley-whittingstall" -> "Hugh Fearnley-Whittingstall"

email


Trims and lowercases the string. Example: "[email protected] " -> "[email protected]"

num


Removes all non-numeric characters. Example: "the price is €30,38" -> "3038"

Note: The struct field will remain a string. No type conversion takes place.

!num


Removes all numbers. Example "39472349D34a34v69e8932747" -> "Dave"

alpha


Removes non-alpha unicode characters. Example: "!@£$%^&'()Hello 1234567890 World+[];\" -> "HelloWorld"

!alpha


Removes alpha unicode characters. Example: "Everything's here but the letters!" -> "' !"

LICENSE

MIT

Owner
Lee Benson
Full-stack developer. JS/Go/C#/Rust. @timberio, https://vector.dev
Lee Benson
Comments
  • Added support for map os string type.

    Added support for map os string type.

    Added conform of maps of string types in the same manner as slices of string types are handled. Respective unit tests were added to cover the new code.

  • [Proposal & PR] add `redact` tag for sensitive string scenario

    [Proposal & PR] add `redact` tag for sensitive string scenario

    I was exploring the solution can help to redact sensitive data from a struct. I realize to use conform as a generic solution probably the best. This addition is simply to transform field to REDACTED if it has the tag. Check the example in _test.go file :)

  • name conform tag strips rest of name with apostrophes

    name conform tag strips rest of name with apostrophes

    Hi There,

    I've noticed that when conforming names with apostrophes, not only will the apostrophe be stripped, but the rest of the name.

    Expected: D'Addario

    Got: D

  • Why not initialization?

    Why not initialization?

    @gislik in https://github.com/leebenson/conform/pull/17 wrote:

    Since no object is instantiated the code is not type-safe, i.e. two thread may try to add sanitizers at the same time resulting in a corrupt state.

    Can we instantiate an object?

    Can we use it like:

    func main() {
      sanitizer := conform.New()
      conform.AddSanitizer(customOne)
    
      sanitizer.String(...)
    }
    

    Why not?

    We're using dependency injection everywhere...

  • support for string array

    support for string array

    Hi,

    Look likes conform won t apply on array of strings, unless i did wrong.

    type NewInput struct {
        Tags        []string `form:"tags[]" validate:"dive,min=1" conform:"trim"`
    }
    func (e EndPoints) HandleForm(c *gin.Context) {
    
      var input NewInput
    
      c.Bind(&input)
    
      conform.Strings(&input)
    
      fmt.Printf("%q\n", input)
      fmt.Printf("%q\n", input.Tags)
    }
    
    curl -D - -d "tags[]=1&tags[]=  " http://localhost:8080/form
    

    yields

    {["1" "  "]}
    ["1" "  "]
    

    Anyways to make that happen ?

  • Slice of int64 errors

    Slice of int64 errors

    It seems like non-string slices are trying to be conformed. I am on version 1.2.2

    type Test struct {
    	Title      string                `json:"test" conform:"trim"`
    	Statuses []int64 `json:"statuses"`
    }
    

    I am getting: reflect.Value.Convert: value of type string cannot be converted to int64

  • Custom sanitizers

    Custom sanitizers

    I made the minimum amount of changes needed to implement custom sanitizers.

    Since no object is instantiated the code is not type-safe, i.e. two thread may try to add sanitizers at the same time resulting in a corrupt state.

  • Fixing a bug where conform panics if a field isn't a struct

    Fixing a bug where conform panics if a field isn't a struct

    Conform was choking on a field in a struct which has a type of []byte.

    The problem was in the recursive Strings method in which reflection is used to figure out the number of fields in a struct. In this edge case the []byte was passed in a recursive walk of the struct which resulted in a panic when calling ift.NumFields().

    This is documented here:

    NumField returns the number of fields in the struct v. It panics if v's Kind is not Struct.

    My solution is to return early in case the reflected variable is not a struct.

    Thank you for your work!

  • Make the email sanitization compliant with rfc

    Make the email sanitization compliant with rfc

    The local part must preserve case, the domain part is lowercased. When supplied with a string containing multiple @s, treat the last @ as the local part/domain part separator.

  • Email transform is wrong

    Email transform is wrong

    The conformatiy for email is wrong as just does an lower(). Because the email addresses [email protected], [email protected], and [email protected] are three different email address according to the RFC and as implemented by many email servers. While case in the domain part does not matter, case in the name part does matter. Also spaces and all weired stuff is allowed if it is escaped or quoted.

  • Conform tag email doesn't work according to the README

    Conform tag email doesn't work according to the README

  • Fixes #38

    Fixes #38

    Add a length check when reading "conform" tags, if length == 0 (<=0) then just ignore the field. This minor change should be enough to fix the problem. Also this is consistent with the original design concept

    Password  string    `schema:"password"`    // <-- no tag? no change
    
  • embedded struct pointer panic

    embedded struct pointer panic

    I always get a panic when my struct has the similar structure as below, here's a minimal example

    package main
    
    import (
    	"github.com/leebenson/conform"
    	"log"
    )
    
    type Child struct {
    	ChildName string `conform:"trim"`
    	Numbers   []uint64
    }
    
    type Parent struct {
    	Child *Child
    }
    
    func main() {
    	child := Child{
    		ChildName: "   child   ",
    		Numbers:   []uint64{1, 2, 3},
    	}
    	parent := Parent{
    		Child: &child,
    	}
    	if err := conform.Strings(&parent); err != nil {
    		log.Println(err)
    	} else {
    		log.Println(*parent.Child)
    	}
    }
    

    output:

    panic: reflect.Value.Convert: value of type string cannot be converted to type uint64
    

    possible fix

    conform.go:243

    if elType.ConvertibleTo(reflect.TypeOf(str)) || elType.ConvertibleTo(reflect.TypeOf(&str)) {
    

    change to

    if elType == reflect.TypeOf(str) || elType == reflect.TypeOf(&str) {
    

    output:

    2022/12/06 22:11:56 {child [1 2 3]}
    

    I might try to make a pr to fix it later, any help or explanation is appreciated, thanks in advance!

  • panic when input struct have a member type of []int

    panic when input struct have a member type of []int

    hi, i got a problem when pass a struct which have an slice of int member, here is the test code below: `package main

    import ( "github.com/leebenson/conform" )

    type UserForm struct { Age []int }

    func main() { input := UserForm{ Age: []int{1}, } conform.Strings(&input) // panic } ` and below is panic info: image

  • Problem with substructs and slice of substruct

    Problem with substructs and slice of substruct

    Hello guys,

    Im trying to implement your code but I have some issues with it.

    In the case I have substruct or slice of substruct, the lib is not working.

    type Request struct {
        SeveralSubRequests []*struct {
    	Foo  string `json:"foo" conform:"trim"`
    	Bar  string `json:"bar" conform:"trim"`
        } `json:"several_sub_requests"`
    }
    
    type Request struct {
        SubRequest *struct {
            Foo  string `json:"foo" conform:"trim"`
            Bar  string `json:"bar" conform:"trim"`
        } `json:"sub_request"`
    }
    
Related tags
Data validation, cleaning and error collection for golang

GoForms - form data validation, cleaning and error reporting The goforms library is a proof-of-concept for a data validation, cleaning and error colle

Jan 14, 2022
Reflectionless data binding for Go's net/http (not actively maintained)
Reflectionless data binding for Go's net/http (not actively maintained)

binding Reflectionless data binding for Go's net/http Features HTTP request data binding Data validation (custom and built-in) Error handling Benefits

Nov 18, 2022
A lightweight go library for parsing form data or json from an http.Request.

Forms Forms is a lightweight, but incredibly useful go library for parsing form data from an http.Request. It supports multipart forms, url-encoded fo

Dec 16, 2022
HTML forms for Golang

HTML forms for Golang Installation: go get github.com/vmihailenco/gforms Example Example: package blog import ( "net/http" "github.com/vmih

Apr 3, 2020
Golang Forms made easy.

Go-FORM-it Description go-form-it makes form creation and handling easy. It allows the creation of form without having to write HTML code or bother to

Aug 20, 2022
CSRF prevention for the Golang Revel framework.

revel-csrf revel-csrf implements Cross-Site Request Forgery (CSRF) attacks prevention for the Revel framework. Code is based on the nosurf package imp

Dec 7, 2022
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

Pagser Pagser inspired by page parser。 Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and str

Dec 13, 2022
Match regex group into go struct using struct tags and automatic parsing

regroup Simple library to match regex expression named groups into go struct using struct tags and automatic parsing Installing go get github.com/oris

Nov 5, 2022
☄ The golang convenient converter supports Database to Struct, SQL to Struct, and JSON to Struct.
☄ The golang convenient converter supports Database to Struct, SQL to Struct, and JSON to Struct.

Gormat - Cross platform gopher tool The golang convenient converter supports Database to Struct, SQL to Struct, and JSON to Struct. 中文说明 Features Data

Dec 20, 2022
AWS Tags Updater - Sync tags with all resources via sheet 🐏🐏

AWS Tags Updater - Sync tags with all resources via sheet ????

Mar 22, 2022
Automatically sets up command line flags based on struct fields and tags.
Automatically sets up command line flags based on struct fields and tags.

Commandeer Commandeer sets up command line flags based on struct fields and tags. Do you... like to develop Go apps as libraries with tiny main packag

Dec 1, 2022
An idiomatic Go (golang) validation package. Supports configurable and extensible validation rules (validators) using normal language constructs instead of error-prone struct tags.

ozzo-validation Description ozzo-validation is a Go package that provides configurable and extensible data validation capabilities. It has the followi

Jan 7, 2023
Align Golang struct tags

Formattag The tool is used to align golang struct's tags. eg.: Before // TestStruct this is a test struct type TestStruct struct { ID stri

Aug 1, 2022
Automatically generate tags for golang struct.

gtag is a command tool that can automatically generate tags for golang struct. Quick start Install gtag into your GOPATH go install github.com/sycki/g

Feb 12, 2022
Automatically generate tags for golang struct.

gotag is a command tool that can automatically generate tags for golang struct. Quick start Install gotag go install github.com/sycki/gotag@latest Pr

Feb 12, 2022
Struct validation using tags

Govalid Use Govalid to validate structs. Documentation For full documentation see pkg.go.dev. Example package main import ( "fmt" "log" "strings"

Dec 6, 2022
A Go library for parsing struct tags from environment variables.

Envconfig Envconfig populates struct field values based on environment variables or arbitrary lookup functions. It supports pre-setting mutations, whi

Jan 2, 2023
Go tool to modify struct field tags
Go tool to modify struct field tags

Go tool to modify/update field tags in structs. gomodifytags makes it easy to update, add or delete the tags in a struct field. You can easily add new tags, update existing tags (such as appending a new key, i.e: db, xml, etc..) or remove existing tags

Jan 1, 2023
goql is a GraphQL client package written in Go. with built-in two-way marshaling support via struct tags.

goql is a GraphQL client package written in Go. with built-in two-way marshaling support via struct tags.

Dec 1, 2022
A linter that handles struct tags.

Tagliatelle A linter that handles struct tags. Supported string casing: camel pascal kebab snake goCamel Respects Go's common initialisms (e.g. HttpRe

Dec 15, 2022