docs/manual/adding-packages-tips.txt - buildroot - Git at Google

 // -*- mode:doc; -*-
 // vim: set syntax=asciidoc:

 === Tips and tricks

 [[package-name-variable-relation]]
 ==== Package name, config entry name and makefile variable relationship

 In Buildroot, there is some relationship between:

 * the _package name_, which is the package directory name (and the
   name of the +*.mk+ file);

 * the config entry name that is declared in the +Config.in+ file;

 * the makefile variable prefix.

 It is mandatory to maintain consistency between these elements,
 using the following rules:

 * the package directory and the +*.mk+ name are the _package name_
   itself (e.g.: +package/foo-bar_boo/foo-bar_boo.mk+);

 * the _make_ target name is the _package name_ itself (e.g.:
   +foo-bar_boo+);

 * the config entry is the upper case _package name_ with `.` and `-`
   characters substituted with `_`, prefixed with +BR2_PACKAGE_+ (e.g.:
   +BR2_PACKAGE_FOO_BAR_BOO+);

 * the +*.mk+ file variable prefix is the upper case _package name_
   with `.` and `-` characters substituted with `_` (e.g.:
   +FOO_BAR_BOO_VERSION+).

 [[check-package]]
 ==== How to check the coding style

 Buildroot provides a script in +utils/check-package+ that checks new or
 changed files for coding style. It is not a complete language validator,
 but it catches many common mistakes. It is meant to run in the actual
 files you created or modified, before creating the patch for submission.

 This script can be used for packages, filesystem makefiles, Config.in
 files, etc. It does not check the files defining the package
 infrastructures and some other files containing similar common code.

 To use it, run the +check-package+ script, by telling which files you
 created or changed:

 ----
 $ ./utils/check-package package/new-package/*
 ----

 If you have the +utils+ directory in your path you can also run:

 ----
 $ cd package/new-package/
 $ check-package *
 ----

 The tool can also be used for packages in a br2-external:

 ----
 $ check-package -b /path/to/br2-ext-tree/package/my-package/*
 ----

 [[testing-package]]
 ==== How to test your package

 Once you have added your new package, it is important that you test it
 under various conditions: does it build for all architectures? Does it
 build with the different C libraries? Does it need threads, NPTL? And
 so on...

 Buildroot runs http://autobuild.buildroot.org/[autobuilders] which
 continuously test random configurations. However, these only build the
 `master` branch of the git tree, and your new fancy package is not yet
 there.

 Buildroot provides a script in +utils/test-pkg+ that uses the same base
 configurations as used by the autobuilders so you can test your package
 in the same conditions.

 First, create a config snippet that contains all the necessary options
 needed to enable your package, but without any architecture or toolchain
 option. For example, let's create a config snippet that just enables
 +libcurl+, without any TLS backend:

 ----
 $ cat libcurl.config
 BR2_PACKAGE_LIBCURL=y
 ----

 If your package needs more configuration options, you can add them to the
 config snippet. For example, here's how you would test +libcurl+ with
 +openssl+ as a TLS backend and the +curl+ program:

 ----
 $ cat libcurl.config
 BR2_PACKAGE_LIBCURL=y
 BR2_PACKAGE_LIBCURL_CURL=y
 BR2_PACKAGE_OPENSSL=y
 ----

 Then run the +test-pkg+ script, by telling it what config snippet to use
 and what package to test:

 ----
 $ ./utils/test-pkg -c libcurl.config -p libcurl
 ----

 By default, +test-pkg+ will build your package against a subset of the
 toolchains used by the autobuilders, which has been selected by the
 Buildroot developers as being the most useful and representative
 subset. If you want to test all toolchains, pass the +-a+ option. Note
 that in any case, internal toolchains are excluded as they take too
 long to build.

 The output lists all toolchains that are tested and the corresponding
 result (excerpt, results are fake):

 ----
 $ ./utils/test-pkg -c libcurl.config -p libcurl
                 armv5-ctng-linux-gnueabi [ 1/11]: OK
               armv7-ctng-linux-gnueabihf [ 2/11]: OK
                         br-aarch64-glibc [ 3/11]: SKIPPED
                            br-arcle-hs38 [ 4/11]: SKIPPED
                             br-arm-basic [ 5/11]: FAILED
                   br-arm-cortex-a9-glibc [ 6/11]: OK
                    br-arm-cortex-a9-musl [ 7/11]: FAILED
                    br-arm-cortex-m4-full [ 8/11]: OK
                              br-arm-full [ 9/11]: OK
                     br-arm-full-nothread [10/11]: FAILED
                       br-arm-full-static [11/11]: OK
 11 builds, 2 skipped, 2 build failed, 1 legal-info failed
 ----

 The results mean:

 * `OK`: the build was successful.
 * `SKIPPED`: one or more configuration options listed in the config
   snippet were not present in the final configuration. This is due to
   options having dependencies not satisfied by the toolchain, such as
   for example a package that +depends on BR2_USE_MMU+ with a noMMU
   toolchain. The missing options are reported in +missing.config+ in
   the output build directory (+~/br-test-pkg/TOOLCHAIN_NAME/+ by
   default).
 * `FAILED`: the build failed. Inspect the +logfile+ file in the output
   build  directory to see what went wrong:
 ** the actual build failed,
 ** the legal-info failed,
 ** one of the preliminary steps (downloading the config file, applying
    the configuration, running `dirclean` for the package) failed.

 When there are failures, you can just re-run the script with the same
 options (after you fixed your package); the script will attempt to
 re-build the package specified with +-p+ for all toolchains, without
 the need to re-build all the dependencies of that package.

 The +test-pkg+ script accepts a few options, for which you can get some
 help by running:

 ----
 $ ./utils/test-pkg -h
 ----

 [[github-download-url]]
 ==== How to add a package from GitHub

 Packages on GitHub often don't have a download area with release tarballs.
 However, it is possible to download tarballs directly from the repository
 on GitHub. As GitHub is known to have changed download mechanisms in the
 past, the 'github' helper function should be used as shown below.

 ------------------------
 # Use a tag or a full commit ID
 FOO_VERSION = 1.0
 FOO_SITE = $(call github,<user>,<package>,v$(FOO_VERSION))
 ------------------------

 .Notes
 - The FOO_VERSION can either be a tag or a commit ID.
 - The tarball name generated by github matches the default one from
   Buildroot (e.g.: +foo-f6fb6654af62045239caed5950bc6c7971965e60.tar.gz+),
   so it is not necessary to specify it in the +.mk+ file.
 - When using a commit ID as version, you should use the full 40 hex characters.
 - When the tag contains a prefix such as +v+ in +v1.0+, then the
   +VERSION+ variable should contain just +1.0+, and the +v+ should be
   added directly in the +SITE+ variable, as illustrated above. This
   ensures that the +VERSION+ variable value can be used to match
   against http://www.release-monitoring.org/[release-monitoring.org]
   results.

 If the package you wish to add does have a release section on GitHub, the
 maintainer may have uploaded a release tarball, or the release may just point
 to the automatically generated tarball from the git tag. If there is a
 release tarball uploaded by the maintainer, we prefer to use that since it
 may be slightly different (e.g. it contains a configure script so we don't
 need to do AUTORECONF).

 You can see on the release page if it's an uploaded tarball or a git tag:

 image::github_hash_mongrel2.png[]

 - If it looks like the image above then it was uploaded by the
   maintainer and you should use that link (in that example:
   'mongrel2-v1.9.2.tar.bz2') to specify +FOO_SITE+, and not use the
   'github' helper.

 - On the other hand, if there's is *only* the "Source code" link, then
   it's an automatically generated tarball and you should use the
   'github' helper function.

 [[gitlab-download-url]]
 ==== How to add a package from Gitlab

 In a similar way to the +github+ macro described in
 xref:github-download-url[], Buildroot also provides the +gitlab+ macro
 to download from Gitlab repositories. It can be used to download
 auto-generated tarballs produced by Gitlab, either for specific tags
 or commits:

 ------------------------
 # Use a tag or a full commit ID
 FOO_VERSION = 1.0
 FOO_SITE = $(call gitlab,<user>,<package>,v$(FOO_VERSION))
 ------------------------

 By default, it will use a +.tar.gz+ tarball, but Gitlab also provides
 +.tar.bz2+ tarballs, so by adding a +<pkg>_SOURCE+ variable, this
 +.tar.bz2+ tarball can be used:

 ------------------------
 # Use a tag or a full commit ID
 FOO_VERSION = 1.0
 FOO_SITE = $(call gitlab,<user>,<package>,v$(FOO_VERSION))
 FOO_SOURCE = foo-$(FOO_VERSION).tar.bz2
 ------------------------

 If there is a specific tarball uploaded by the upstream developers in
 +https://gitlab.com/<project>/releases/+, do not use this macro, but
 rather use directly the link to the tarball.
	// -- mode:doc; --
	// vim: set syntax=asciidoc:

	=== Tips and tricks

	[[package-name-variable-relation]]
	==== Package name, config entry name and makefile variable relationship

	In Buildroot, there is some relationship between:

	* the _package name_, which is the package directory name (and the
	name of the +*.mk+ file);

	* the config entry name that is declared in the +Config.in+ file;

	* the makefile variable prefix.

	It is mandatory to maintain consistency between these elements,
	using the following rules:

	* the package directory and the +*.mk+ name are the _package name_
	itself (e.g.: +package/foo-bar_boo/foo-bar_boo.mk+);

	* the _make_ target name is the _package name_ itself (e.g.:
	+foo-bar_boo+);

	* the config entry is the upper case _package name_ with `.` and `-`
	characters substituted with `_`, prefixed with +BR2_PACKAGE_+ (e.g.:
	+BR2_PACKAGE_FOO_BAR_BOO+);

	* the +*.mk+ file variable prefix is the upper case _package name_
	with `.` and `-` characters substituted with `_` (e.g.:
	+FOO_BAR_BOO_VERSION+).

	[[check-package]]
	==== How to check the coding style

	Buildroot provides a script in +utils/check-package+ that checks new or
	changed files for coding style. It is not a complete language validator,
	but it catches many common mistakes. It is meant to run in the actual
	files you created or modified, before creating the patch for submission.

	This script can be used for packages, filesystem makefiles, Config.in
	files, etc. It does not check the files defining the package
	infrastructures and some other files containing similar common code.

	To use it, run the +check-package+ script, by telling which files you
	created or changed:

	----
	$ ./utils/check-package package/new-package/*
	----

	If you have the +utils+ directory in your path you can also run:

	----
	$ cd package/new-package/
	$ check-package *
	----

	The tool can also be used for packages in a br2-external:

	----
	$ check-package -b /path/to/br2-ext-tree/package/my-package/*
	----

	[[testing-package]]
	==== How to test your package

	Once you have added your new package, it is important that you test it
	under various conditions: does it build for all architectures? Does it
	build with the different C libraries? Does it need threads, NPTL? And
	so on...

	Buildroot runs http://autobuild.buildroot.org/[autobuilders] which
	continuously test random configurations. However, these only build the
	`master` branch of the git tree, and your new fancy package is not yet
	there.

	Buildroot provides a script in +utils/test-pkg+ that uses the same base
	configurations as used by the autobuilders so you can test your package
	in the same conditions.

	First, create a config snippet that contains all the necessary options
	needed to enable your package, but without any architecture or toolchain
	option. For example, let's create a config snippet that just enables
	+libcurl+, without any TLS backend:

	----
	$ cat libcurl.config
	BR2_PACKAGE_LIBCURL=y
	----

	If your package needs more configuration options, you can add them to the
	config snippet. For example, here's how you would test +libcurl+ with
	+openssl+ as a TLS backend and the +curl+ program:

	----
	$ cat libcurl.config
	BR2_PACKAGE_LIBCURL=y
	BR2_PACKAGE_LIBCURL_CURL=y
	BR2_PACKAGE_OPENSSL=y
	----

	Then run the +test-pkg+ script, by telling it what config snippet to use
	and what package to test:

	----
	$ ./utils/test-pkg -c libcurl.config -p libcurl
	----

	By default, +test-pkg+ will build your package against a subset of the
	toolchains used by the autobuilders, which has been selected by the
	Buildroot developers as being the most useful and representative
	subset. If you want to test all toolchains, pass the +-a+ option. Note
	that in any case, internal toolchains are excluded as they take too
	long to build.

	The output lists all toolchains that are tested and the corresponding
	result (excerpt, results are fake):

	----
	$ ./utils/test-pkg -c libcurl.config -p libcurl
	armv5-ctng-linux-gnueabi [ 1/11]: OK
	armv7-ctng-linux-gnueabihf [ 2/11]: OK
	br-aarch64-glibc [ 3/11]: SKIPPED
	br-arcle-hs38 [ 4/11]: SKIPPED
	br-arm-basic [ 5/11]: FAILED
	br-arm-cortex-a9-glibc [ 6/11]: OK
	br-arm-cortex-a9-musl [ 7/11]: FAILED
	br-arm-cortex-m4-full [ 8/11]: OK
	br-arm-full [ 9/11]: OK
	br-arm-full-nothread [10/11]: FAILED
	br-arm-full-static [11/11]: OK
	11 builds, 2 skipped, 2 build failed, 1 legal-info failed
	----

	The results mean:

	* `OK`: the build was successful.
	* `SKIPPED`: one or more configuration options listed in the config
	snippet were not present in the final configuration. This is due to
	options having dependencies not satisfied by the toolchain, such as
	for example a package that +depends on BR2_USE_MMU+ with a noMMU
	toolchain. The missing options are reported in +missing.config+ in
	the output build directory (+~/br-test-pkg/TOOLCHAIN_NAME/+ by
	default).
	* `FAILED`: the build failed. Inspect the +logfile+ file in the output
	build directory to see what went wrong:
	** the actual build failed,
	** the legal-info failed,
	** one of the preliminary steps (downloading the config file, applying
	the configuration, running `dirclean` for the package) failed.

	When there are failures, you can just re-run the script with the same
	options (after you fixed your package); the script will attempt to
	re-build the package specified with +-p+ for all toolchains, without
	the need to re-build all the dependencies of that package.

	The +test-pkg+ script accepts a few options, for which you can get some
	help by running:

	----
	$ ./utils/test-pkg -h
	----

	[[github-download-url]]
	==== How to add a package from GitHub

	Packages on GitHub often don't have a download area with release tarballs.
	However, it is possible to download tarballs directly from the repository
	on GitHub. As GitHub is known to have changed download mechanisms in the
	past, the 'github' helper function should be used as shown below.

	------------------------
	# Use a tag or a full commit ID
	FOO_VERSION = 1.0
	FOO_SITE = $(call github,<user>,<package>,v$(FOO_VERSION))
	------------------------

	.Notes
	- The FOO_VERSION can either be a tag or a commit ID.
	- The tarball name generated by github matches the default one from
	Buildroot (e.g.: +foo-f6fb6654af62045239caed5950bc6c7971965e60.tar.gz+),
	so it is not necessary to specify it in the +.mk+ file.
	- When using a commit ID as version, you should use the full 40 hex characters.
	- When the tag contains a prefix such as +v+ in +v1.0+, then the
	+VERSION+ variable should contain just +1.0+, and the +v+ should be
	added directly in the +SITE+ variable, as illustrated above. This
	ensures that the +VERSION+ variable value can be used to match
	against http://www.release-monitoring.org/[release-monitoring.org]
	results.

	If the package you wish to add does have a release section on GitHub, the
	maintainer may have uploaded a release tarball, or the release may just point
	to the automatically generated tarball from the git tag. If there is a
	release tarball uploaded by the maintainer, we prefer to use that since it
	may be slightly different (e.g. it contains a configure script so we don't
	need to do AUTORECONF).

	You can see on the release page if it's an uploaded tarball or a git tag:

	image::github_hash_mongrel2.png[]

	- If it looks like the image above then it was uploaded by the
	maintainer and you should use that link (in that example:
	'mongrel2-v1.9.2.tar.bz2') to specify +FOO_SITE+, and not use the
	'github' helper.

	- On the other hand, if there's is only the "Source code" link, then
	it's an automatically generated tarball and you should use the
	'github' helper function.

	[[gitlab-download-url]]
	==== How to add a package from Gitlab

	In a similar way to the +github+ macro described in
	xref:github-download-url[], Buildroot also provides the +gitlab+ macro
	to download from Gitlab repositories. It can be used to download
	auto-generated tarballs produced by Gitlab, either for specific tags
	or commits:

	------------------------
	# Use a tag or a full commit ID
	FOO_VERSION = 1.0
	FOO_SITE = $(call gitlab,<user>,<package>,v$(FOO_VERSION))
	------------------------

	By default, it will use a +.tar.gz+ tarball, but Gitlab also provides
	+.tar.bz2+ tarballs, so by adding a +<pkg>_SOURCE+ variable, this
	+.tar.bz2+ tarball can be used:

	------------------------
	# Use a tag or a full commit ID
	FOO_VERSION = 1.0
	FOO_SITE = $(call gitlab,<user>,<package>,v$(FOO_VERSION))
	FOO_SOURCE = foo-$(FOO_VERSION).tar.bz2
	------------------------

	If there is a specific tarball uploaded by the upstream developers in
	+https://gitlab.com/<project>/releases/+, do not use this macro, but
	rather use directly the link to the tarball.