Using these targets on a pkgbase system will cause the installed system
to become out of sync with the package database, which is almost certain
to cause issues the next time pkg(8) is used.
To prevent users doing this accidentally, disallow install* if we detect
that FreeBSD-runtime is installed in the target. The check can be
overridden with DESTDIR=/ for users who are sure they want to do this.
MFC after: 3 days
Reviewed by: emaste
Sponsored by: https://www.patreon.com/bsdivy
Differential Revision: https://reviews.freebsd.org/D52879
The FreeBSD-base pkg repository is currrently dominated by a handful of
large packages: Out of a compressed repository size of ~960 MB, about
160 MB is taken up by the FreeBSD-src package, 128 MB (on amd64) is
used by the FreeBSD-kernel-generic-dbg package, and 91 MB is used by
the FreeBSD-src-sys package. Consequently, running 'make packages -jN'
provides less benefit than one might hope, as most of the packages
finish building quickly, ultimately leaving the FreeBSD-src package
building by itself for a couple minutes while all the other CPUs are
idle.
Pass -T${PKG_CTHREADS} to the 'pkg create' commands (with a default
of -T0) in order to instruct pkg's zstd compression to use multiple
threads.
Testing on an EC2 r7i.48xlarge instance with -j192, this reduces the
time taken by 'make packages' from 6m17s to 1m39s; package creation
time itself (excluding the initial process of installing into world
and kernel staging directories) dropped from 5m37s to 59s.
Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53053
To build set-src, we first need to build the source packages. Add a
.ORDER to ensure this happens. Otherwise, in a parallel build, sets
might be built before the src-* packages have finished building, and
set-src will be mysteriously missing.
MFC after: 3 seconds
Reported by: cperciva
Actually diagnosed by: jrtc27
One-line fix by: ivy
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D53076
dtb.plist is generated by mtree-to-plist when it processes the kernel
metalog, so we shouldn't try to create dtb packages until that's been
done. Add another level of indirection to the create-kernel-packages
target to organize all of the kernel flavors under a single target that
is ordered before the create-dtb-packages target.
Note that we may later refine these targets after further pkgbase
review; this is simply a maybe-final stop-gap that's been tested to
fix a known problem.
MFC after: 37 seconds (pkgbase fix for 15.0 builds)
Reviewed and tested by: cperciva
Differential Revision: https://reviews.freebsd.org/D53017
The general problem is that we might have a somewhat complicated
dependency tree depending on bootstrap version requirements. We could
document when multiple bootstrap tools might have a shared dependency
and be careful to add them only once to the list, but that is a little
more fragile- particularly if we purge some bootstrap tools and need to
re-work the logic a little bit.
Just avoid redefining the build commands as we're iterating over the
list so that we can keep the actual requirements intact.
PR: 289997
Reported by: dhw, others
Reviewed by: imp
Fixes: 151bd3516b ("flua: support our flua modules in [...]")
Differential Revision: https://reviews.freebsd.org/D52904
The .if defined(_MKSHOWCONFIG) covered an unusually large area, so it
should have not been removed in the last commit. I must have tested in
the wrong tree before pushing...
FixeS: 28b858f505
Sponsored by: Netflix
We don't need this, and we don't use this. It's left over from the svn
days. We stopped supporting svn as a project entirely when 12.x went
EOL. And VCS_REVSION isn't in any current ucl file or anywhere else in
the tree.
Sponsored by: Netflix
Reviewed by : kevans, brd
Differential Revision: https://reviews.freebsd.org/D52912
This version builds every module into the flua binary itself, since all
of the bootstrap tools are built -DNO_SHARED. As a result, we also
cannot dlsym(), so we can't really discover the names of our newly
builtin modules. Instead, just build out a linker set with all of our
luaopen_*() functions to register everything up-front.
Building in all of the modules isn't strictly necessary, but it means
that we have an example of how to add a bootstrap module everywhere you
go and one doesn't need to consider whether bootstrap flua can use a
module when writing scripts. On my build machine, the consequence on
our binary size is an increase from around 1.6M -> 1.9M, which isn't
really that bad.
.lua modules can install into their usual path below $WORLDTMP/legacy
and we'll pick them up automagically by way of the ctor that sets up
LUA_PATH early on.
This re-lands bootstrap module support with a more sensible subset, and
after having verified that it cross-builds fine on macOS and Linux -- we
cannot do libfreebsd on !FreeBSD because it's more system header
dependant. We also need to bootstrap libmd to bring in libhash, and
libucl + libyaml.
Reviewed by: bapt, emaste (both previous version)
Differential Revision: https://reviews.freebsd.org/D51890
These targets are used to produce legacy dist sets for install media and
now always use NO_ROOT mode. Extend existing logic that forces NO_ROOT
mode to these cases to ensure they do not run in the wrong mode.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D50551
When NO_INSTALLKERNEL and NO_INSTALLEXTRAKERNELS are set, or when
BUILDKERNELS is empty, we can fail to define the create-kernel-packages
make target, which we call unconditionally. So if do not have a kernel
staging dir, go ahead and provide an empty create-kernel-packages so
that when it's used elsewhere, the build succeeds.
This lets me build a tree and package it when I don't have a kernel, for
jails and other similar environments. There's no change if any kernels
are built, since that will provide the necessary make target that does
useful things for that case.
Sponsored by: Netflix
package-pkg (via make-pkg-package.sh) passes CONFIGURE_ARGS to make when
building ports-mgmt/pkg, which overrides the port's default configure
args that are supposed to set --prefix. This means that pkg is always
built with the default prefix of /usr/local, which then fails when ports
tries to package it from LOCALBASE.
Work around this by explicitly adding --prefix to CONFIGURE_ARGS.
MFC after: 1 day
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D52634
Development versions of pkg have a version like "2.3.0-d8bbde1c3-dirty",
which is not handled by PKG_BIN_VERSION, so PKG_EXT ends up being "tar"
and this breaks update-packages.
Fix the PKG_BIN_VERSION logic, and while here, remove backwards compat
for pkg versions prior to 2.0, which no one should be using to build
main or stable/15. This means nothing is left to use PKG_BIN_VERSION,
but continue to set it, since we might need it again in the future.
MFC after: 1 day
Reviewed by: manu, kevans
Differential Revision: https://reviews.freebsd.org/D52666
In release builds we want to build pkg from the ports tree for reasons
mentioned in a previous commit; but non-release pkgbase builds may be
just fine with installing pkg from pkg.freebsd.org.
Gate it behind BOOTSTRAP_PKG_FROM_PORTS rather than building form ports
any time we have a ports tree available.
While I'm here, make the pkgbase-repo target in release/Makefile
create the temporary directories we need for building.
Reviewed by: ivy
MFC after: 6 hours
Sponsored by: https://www.patreon.com/cperciva
Fixes: 2c06b82443 ("pkgbootstrap: Build pkg if PORTSDIR exists")
We need to put pkg into the pkgbase repository for release builds, but
not for non-release pkgbase builds. Gate the contents of package-pkg
behind INCLUDE_PKG_IN_PKGBASE_REPO=YES and set that when building
releases.
Reviewed by: ivy
MFC after: 6 hours
Sponsored by: https://www.patreon.com/cperciva
If we don't already have the pkg tool installed, we need to install it
before we can build packages for pkgbase. (This arises in release
builds which run in clean chroots; most users will have pkg installed
already.) Installing it from pkg.freebsd.org via "pkg bootstrap" runs
into two problems: First, it's an external dependency with security
implications; and second, there's a bootstrapping problem on new major
versions where pkg.freebsd.org does not have packages yet.
Instead, if we have a ports tree, use it to build pkg ourselves.
With hat: re
MFC after: 6 hours
Sponsored by: https://www.patreon.com/cperciva
At sufficiently high levels of build parallelism we can try to create
the pkg before any of the other pkgbase-creation steps have taken
care of the bootstrap.
Fixes: 6b5adf3387 ("Include a pkg package in the pkgbase repo")
MFC after: 12 hours
Sponsored by: https://www.patreon.com/cperciva
It is essential that users be able to install the FreeBSD base system
from release media and have all the bits needed to update the FreeBSD
base system without touching the ports tree or pkg.freebsd.org. To
that end, resurrect (and heavily rewrite) the make-pkg-package.sh
script and hook it into the create-packages target; if /usr/ports
exists when building pkgbase packages, we'll also (cross)build pkg
and include it in the repository.
Scripting for actually installing this package as part of the FreeBSD
installation process to come later, but I wanted this in the tree in
time to test it in this week's snapshots.
MFC after: 1 day
Sponsored by: https://www.patreon.com/cperciva
Add several metapackages which contain no files themselves, but depend
on other packages. This allows the user to install a system by using
a set like "FreeBSD-set-minimal" instead of having to know which
specific packages are required, and it means if we add more package
in the future, existing installations will get them automatically as
long as they were installed using a set.
The defined sets are significantly less granular than the actual
packages; the assumption is that users who want a very specific set of
packages installed can do that manually, while sets are intended more
for typical users who just want to install FreeBSD.
The following sets are defined:
* minimal: the set of packages required to boot the system and bring
up a multi-user UNIX system. This includes hardware support,
networking (wireless, DHCP), basic functionality like syslogd, cron
and periodic. It does not include a kernel, because the kernels are
quite large and the user might want to use a custom kernel.
sendmail and DMA are not included in minimal. Since we provide two
MTAs, the user should select which one they want, or install one from
ports. sshd is not included because it's not required for the base
system, and the user might want to install it from ports.
Some other significant subsystems (e.g. NFS, Kerberos) are also not
included.
The minimal set does not include any libraries itself, since we rely
on dependencies to pull these in. This implies it doesn't include
library manpages, which is fine, since users won't want manpages for
libraries unless they've also installed the development packages.
* devel: the compiler toolchain for building software, along with all
"-dev" packages which include header files and static libraries.
* lib32: 32-bit compatibility packages. This includes both runtime
and development packages.
* kernels: the kernel packages. For freebsd.org builds, this will be
GENERIC, GENERIC-DEBUG, etc., for custom builds it will be whatever
was set in $KERNCONF.
* base: everything else; this includes the entire base system.
For each set, also generate a <setname>-dbg set containing the debug
packages for the set.
The sets are built along with the rest of the packages using a new
script called create-sets.sh, which examines the "set" annotation in
each package and puts it in the appropriate set. This is in
anticipation of the later "groups" feature appearing in pkg itself, at
which point we can simply replace the set annotation with the group.
MFC after: 3 days
Reviewed by: imp, bapt
Differential Revision: https://reviews.freebsd.org/D52412
Buildworld failed when objcopy tried to overwrite itself, with `objcopy:
open objcopy failed: Text file busy`. The PATH ended up with `::`,
effectively including the current directory, and we found the wrong
objcopy.
PR: 261215
Sponsored by: The FreeBSD Foundation
Fix list-old-dirs to not generate empty line at the end of the
output if OLD_DIRS happens to have a space at the end of the
last word of the output before the final \n. Then that space
is turned into \n and we end up with a blank line.
Futhermore this gets converted into a "/" i.e. root fs when
calling check-old-dirs.
This is the regression since a8267ecc3d.
Reviewed by: emaste
Approved by: emaste
Sponsored by: Sippy Software, Inc.
Differential Revision: https://reviews.freebsd.org/D52153
MFC After: 3 days
As in bsd.lib.mk and bsd.prog.mk, we need to remap paths in the kernel
and in kernel modules when reproducible builds are configured. Add
directives to kern.post.mk and kmod.mk to remap the src, obj and sysroot
dirs.
For the sysroot, we need to pass the value of the SYSROOT variable to
the kernel makefiles, which involves changing Makefile.inc1. In so
doing I reverted a portion of commit 35fb812adfc21; I can't see any harm
in passing the SYSROOT variable down.
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D50953
For MIT Kerberos, MK_GSSAPI has no meaning: GSSAPI is a required part of
Kerberos and is always built if MK_KERBEROS is enabled. Backport this
behaviour to Heimdal so it works the same way.
While here, change Heimdal's libcom_err and compile_et to be selected by
MK_KERBEROS, not MK_KERBEROS_SUPPORT, since these are part of Kerberos
and third-party users might need it even if Kerberos support is disabled
in the base system. This means MK_KERBEROS_SUPPORT installs the same
files with both MIT and Heimdal.
Reviewed by: cy
Differential Revision: https://reviews.freebsd.org/D51859
Some platforms, like macOS, do not expose headers for the system's
libcrypto for public consumption. libcrypto is relatively heavy and
needs to know, e.g., the host system's endianness, so we scope the build
down to macOS where OpenSSL headers are known to not be present and we
can be reasonably certain that most of the systems today that would be
cross-building are little endian.
We still don't bother if building WITHOUT_OPENSSL since the end result
is expected to be used by OpenSSL, but perhaps we could revisit that
independently in case one, e.g., brings their own implementation.
Reported by: jrtc27
Reviewed by: jrtc27, ngie
Fixes: c340ef28fd ("certctl: Reimplement in C")
Differential Revision: https://reviews.freebsd.org/D51935
The variable's definition was removed with texinfo in
2d2813618c, but the reference to the
variable was left behind.
This is effectively a no-op cleanup.
MFC after: 3 days
Notable changes include:
* We no longer forget manually untrusted certificates when rehashing.
* Rehash will now scan the existing directory and progressively replace
its contents with those of the new trust store. The trust store as a
whole is not replaced atomically, but each file within it is.
* We no longer attempt to link to the original files, but we don't copy
them either. Instead, we write each certificate out in its minimal
form.
* We now generate a trust bundle in addition to the hashed diretory.
This also contains only the minimal DER form of each certificate.
This allows e.g. Unbound to preload the bundle before chrooting.
* The C version is approximately two orders of magnitude faster than the
sh version, with rehash taking ~100 ms vs ~5-25 s depending on whether
ca_root_nss is installed.
* We now also have tests.
Reviewed by: kevans, markj
Differential Revision: https://reviews.freebsd.org/D42320
Differential Revision: https://reviews.freebsd.org/D51896
The reimplementation of certctl, while much needed, broke the release
build and 72 hours later corrections are still under review (D51896).
This revert should be reverted once that is ready to land; I just need
this out of the tree temporarily because breakage is interfering with
release engineering for the upcoming 15.0-RELEASE.
Unsquashed reversions:
Revert "etc: add missing mtree entry for certctl tests"
This reverts commit f751757259.
Revert "certctl: Fix bootstrap build"
This reverts commit c989e3cc3d.
Revert "certctl: Reimplement in C"
This reverts commit 81d8827ad8.
With hat: re@
Idea and file format shamelessly stolen from CheriBSD, but reimplemented
in terms of the standard FreeBSD build system. We'll use this in some
events that call for a deeper cleansing, typically reserved for
situations where the dependencies are too complicated to unwind. This
notably does not preclude us from doing separate cleansing of world for
specific src.conf(5) knob changes that would require a rebuild.
In the FreeBSD version, we either stamp the OBJTOP we're cleaning
(bootstrap or the full OBJDIR) with the current epoch and bail out for
unstamped objdirs, or we compare and either `rm -rf` or `cleandir` as
necessary.
Reviewed by: brooks, des
Differential Revision: https://reviews.freebsd.org/D51848
Notable changes include:
* We no longer forget manually untrusted certificates when rehashing.
* Rehash will now scan the existing directory and progressively replace
its contents with those of the new trust store. The trust store as a
whole is not replaced atomically, but each file within it is.
* We no longer attempt to link to the original files, but we don't copy
them either. Instead, we write each certificate out in its minimal
form.
* We now generate a trust bundle in addition to the hashed diretory.
This also contains only the minimal DER form of each certificate.
* The C version is approximately two orders of magnitude faster than the
sh version, with rehash taking ~100 ms vs ~5-25 s depending on whether
ca_root_nss is installed.
* The DISTBASE concept has been dropped; the same effect can be achieved
by adjusting DESTDIR.
* We now also have rudimentary tests.
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D42320
lib/libgssapi is based on Heimdal. As on Linux systems, the MIT
libgssapi_krb5 replaces it. With both gssapi libraries and header files
installed results in broken buildworld (gssd) and ports that will not
build without modifications to support the MIT gssapi in an alternate
location.
73ed0c7992 removed the MIT GSSAPI headers from /usr/include. Apps using
MIT KRB5 gssapi functions and structures will fail to build without this
patch.
This patch includes a temporary patch to usr.sbin/gssd to allow it
to build with this patch. rmacklem@ has a patch for this and for
kgssapi that uses this patch to resolve kgssapi issues for NFS with
Kerberos.
This patch is an updated version of D51661 to allow it to build following
additional patchs to the tree.
This should have been implmented with 7e35117eb0.
Fixes: 7e35117eb0, 73ed0c7992
Differential Revision: https://reviews.freebsd.org/D51661
libedit breaks the bootstrap on MacOS and Linux.
Activate libedit only for the regular build not for the bootstrap
tools
While here fix the definition of the dependency chain between
libkrb5ss and libedit (and libtinfow) via src.libnames.mk
Remove a local patch to find the readline compatible header and
find them via proper CFLAGS.
Modify template.ucl to require that the "all" UCL file exists for every
package. This prevents packages being added accidentally, and ensures
each package has as least a perfunctory comment and description.
Add missing UCL files for all existing packages.
Modify special packages created by Makefile.inc1 to pass the appropriate
parameters to generate-ucl to find their UCL files.
Reviewed by: des, bapt
Approved by: des (mentor)
Differential Revision: https://reviews.freebsd.org/D50281
In bootstrap-tools we search for some tools to copy into the obj tree.
If the tool is missing in the host environment then the '[ ! -e ... ]'
check will fail as which will print '<tool> not found'.
There is an issue that which will also return an error so we never get
to the check and don't print the error message. As which outputs an
invalid path on failure we can just add || true to ensure make
continues.
Reviewed by: kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D51075
Otherwise timestamps in the underlying tar files are not reproducible.
I believe this was a simple omission, other package targets specify the
timestamp this way.
Reviewed by: bapt
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D51066
Revert b44147bb12. It doesn't work right since it doesn't bootstrap
things quite right. It doesn't look trivial to fix, so revert until a
proper solution is done (though maybe that solution is just do `make
update-packages` and not document/support these other targets).
Suggested by: bapt, emaste
Sponsored by: Netflix
This was an attempt to fix b44147bb12, but it's still not right. Per
conversation on irc, revert it and the other commit so we can redo them
correctly.
Suggested by: bapt, kevans
After b44147bb12 there is no dependence
relationshop between create-packages and real-update-packages, which
results on the package building cluster in a race condition
real-update-packages tries to open a package which is being created by
the create-packages target and make the whole build fail.
Ensure create-packages and real-update-packages is sequential
Add tests for MK_MITKRB5. If "yes" build MIT KRB5. If "no" build Heimdal.
The default is MK_MITKRB5 = no, added by "krb5: Add build plumbing".
At some point we will change the default to MK_MITKRB5 = yes. A ports
exp-run will need to be successfully run first.
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D50815
To perform an incremental update of a pkgbase repo, you would
call 'make update-packages', which will stage, create, and
incrementally choose newer package versions to sign as part
of a pkg repo. However, this forces you to stage the kernel
and source packages along with the world packages. For a
jail-only installation of FreeBSD, these packages are generally
not required.
This patch separates the 'update-packages' target from the
'real-update-packages' target to allow choosing world, kernel,
and/or source individually at the command line like so:
Jail-only installation:
1. make -j`nproc` buildworld
2. make PKG_VERSION='15.snap<date>' \
stage-packages-world create-packages-world \
real-update-packages
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/1445
Closes: https://github.com/freebsd/freebsd-src/pull/1445
When switching between compressed and uncompressed manual pages, the
other type is left behind as divots. Compensate by removing the
now-unwanted man pages automatically.
Reviewed by: imp, jhb
Pull Request: https://github.com/freebsd/freebsd-src/pull/1295
Commit 1cbb58886a (shipped in 12.0.0) removed all lint infrastructure.
A bunch of NO_LINT definitions remained (perhaps as a bootstrapping
measture). Remove them.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D50704
As of commit 41adc5f29b ("release: Always use NO_ROOT for distribute*
and package*") this packagekernel code path is never used, so remove it.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D50361
Commit 49bc071f40 ("nsswitch.conf: Avoid modification after
installation") changes handling of nsswitch.conf such that we make a
copy in the objdir during a build. Historically, the in-tree
nsswitch.conf lived under etc.
The buildetc target and its copy set SUBDIR_OVERRIDE=etc when building
the object tree, but I think this isn't right when conf files are
scattered around the src tree. If any of them require non-trivial
processing, they'll get skipped during buildetc, and then some
build-time commands may run during installetc. In the linked PR, this
fails because the src tree is mounted read-only and no objdir was
created during buildetc.
Remove the SUBDIR_OVERRIDE for the _obj target, and build the
buildconfig target across the tree.
PR: 286072
Fixes: 49bc071f40 ("nsswitch.conf: Avoid modification after installation")
Reviewed by: brooks, dim
Tested by: dim, Alastair Hogge <agh@riseup.net>
Differential Revision: https://reviews.freebsd.org/D49960
When WITH_CLANG_BOOTSTRAP and WITHOUT_CLANG are both set, the
cross-tools stage does not build a cross clang binary. This is because
the Makefile in usr.bin/clang checks for WITHOUT_CLANG, and skips
building the binary.
To fix this, ensure that WITH_CLANG is set for the cross-tools phase
whenever WITH_CLANG_BOOTSTRAP is set. While here, skip using the
Makefile in usr.bin/clang, and directly use the Makefile in
usr.bin/clang/clang instead.
PR: 286154
Reported by: avg
Reviewed by: avg, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D49886
ELF Tool Chain's objcopy-equivalent is called elfcopy. Restore the
upstream name in our build infrastructure to make it more clear where
different binary utility components come from.
Reviewed by: brooks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D49784
Lexi Winter