The Whonix Makefile currently prevents build of Qubes template since it interferes with qubes-builder directives.
I have not yet identified the offending code within the Makefile or a possible solution as of yet.
As a stop-gap measure, I have deleted the Makefile to allow a build, which will cause a build-error if the Whonix repo is not committed. It would be nice to have an option to skip this test when building.
Will provide further details once I have Whonix 10 building and tested.
All packages or just one?
If you post the xtrace, I'll find out which function causes this.
If it's just one package, we can easily add a package specific override (the override mechanism I explained in T168). Otherwise if multiple packages are involved, we can just patch the genmkfile package.
By the way... If it's about qubes-whonix... Can you merge my changes please?
https://github.com/Whonix/qubes-whonix
Updated genmkfile to version 1.5.
I have merged your changes with the v9.6.6 version of qubes-whonix which is currently the master and also the branch with the name of 9.6.6. 9.6.6 also contains the fixes for immutable files, and the newer whonix-setup-wizard.
These changes were also merged to the 10.0.1 branch. Currently all 3 branches contain identical code being the version numbers being the only difference as of right now.
So far build seems fine for both gateway and workstation.
I created a fix with the qubes-builder template code that builds Whonix to fix the error of the task topic by exporting GENMKFILE_BOOTSTRAP to an empty marker file so the Whonix makefile-full will not be included.
It would be useful for me if you could change the main Whonix Makefile's last line from:
include $(GENMKFILE_BOOTSTRAP)/makefile-full to -include $(GENMKFILE_BOOTSTRAP)/makefile-full
This way I will not actually have to need to have a makefile-full marker file and can just set a fake path that will fail. When you add the minus - sign in front of a make file statement, that indicates to make not to fail if it can not find or load the file.
Just this single one https://github.com/Whonix/Whonix/blob/master/Makefile ?
If it would fail open, I guess that would be confusing.
Why does qubes-builder not like it? I am interested to make it real compatible, extensible, easy, clean and whatnot.
But yes, that main Whonix Makefile has room for customization. We can make this real clean and awesome.
From include $(GENMKFILE_BOOTSTRAP)/makefile-full would it help to make the latter part makefile-full configurable by an environment variable also? Too bad I didn't think of that earlier.
But that would be a hack. Because then you'd set GENMKFILE_ environment variables, but did something completely different. There could also be something like GENMKFILE_CUSTOM and if set, it does your custom stuff and ignores the rest.
Or perhaps there should also be a clean [environment variables?] way to - earlier and/or later - to source your own Makefile?
Yes, you are referencing the correct file. It is the only file that needs to be edited and only needs the additional -' inserted before the include` statement.
It fails for qubes-builder since qubes-builder also includes or tests for the existence of certain 'targets' of all Makefiles that are included as a COMPONENT. The Makefile you are including must have a conflicting target; or has a target that is being called by qubes-builder that is not intended to be called by Whonix. I include Whonix as a component to allow its source code to automatically be downloaded and verified ans well as to prompt the user to allow importing the Whonix key.
Yes, the later part could also be configurable.
Another les hackish way to accomplish the goal would be to instead add a conditional to prevent the include statement from executing if some ENV var is set. Something like the following:
INCLUDE_MKFILE ?= 1 GENMKFILE_NAME ?= makefile-full GENMKFILE_BOOTSTRAP ?= ./packages/genmkfile/usr/share/genmkfile GENMKFILE_PATH ?= $(GENMKFILE_BOOTSTRAP) GENMKFILE_ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))) #You could use the following in lieu of exporting each var individually #.EXPORT_ALL_VARIABLES: export GENMKFILE_BOOTSTRAP export GENMKFILE_PATH export GENMKFILE_ROOT_DIR # If the ENV var INCLUDE_MKFILE == 0, then makefile will NOT be included ifeq ($(INCLUDE_MKFILE), 1) include $(GENMKFILE_BOOTSTRAP)/$(GENMKFILE_NAME) endif
Yeah. (Untested, but...) I like that!
There are two specific Makefiles (can probably not be avoided)
And the generic one https://github.com/Whonix/genmkfile/blob/master/usr/share/genmkfile/Makefile that all the other packages are currently using.
I would like to fix this once and for all, if that's possible, at least for most cases, a long time.
See the following. Roughly tested. But you most likely know what I am up to? Hopefully you like fixing it? I guess it's not that hard if you speak make?
#!/usr/bin/make -f ## Copyright (C) 2012 - 2014 Patrick Schleizer <adrelanos@riseup.net> ## See the file COPYING for copying conditions. ## Bootstrapping genmkfile. Using genmkfile to build genmkfile. ## Using genmkfile without having genmkfile available as a build dependency. GENMKFILE_NAME ?= makefile-full GENMKFILE_BOOTSTRAP ?= ./packages/genmkfile/usr/share/genmkfile GENMKFILE_PATH ?= $(GENMKFILE_BOOTSTRAP) GENMKFILE_ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))) GENMKFILE_INCLUDE_FILE_MAIN ?= $(GENMKFILE_PATH)/$(GENMKFILE_NAME) export GENMKFILE_NAME export GENMKFILE_BOOTSTRAP export GENMKFILE_PATH export GENMKFILE_ROOT_DIR export GENMKFILE_INCLUDE_FILE_MAIN ifdef GENMKFILE_INCLUDE_FILE_PRE include $(GENMKFILE_INCLUDE_FILE_PRE) endif ifneq ($(GENMKFILE_INCLUDE_FILE_MAIN),0) include $(GENMKFILE_INCLUDE_FILE_MAIN) endif ifdef GENMKFILE_INCLUDE_FILE_POST include $(GENMKFILE_INCLUDE_FILE_POST) endif
Changes:
Could also become genmkfile - Makefile - version 1.6.
Questions:
I tried to reply from email; guess it does not work that way?
Statements like ifneq ($(GENMKFILE_INCLUDE_FILE_MAIN),0) are pretty much the way to do what we are trying to do with make. Lots of things bug me about make, but it is very powerful.
I would think the make file should exit with an error if you are including a file that does not exist, unless it is preceded with a -. Make sure that there are no tabs in the statement. I guess you could also test if the file exists as well like:
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_PRE))) $(error GENMKFILE_INCLUDE_FILE_PRE $(GENMKFILE_INCLUDE_FILE_PRE) does not exist!) endif
One thing I always do is add an about target into all my make files:
.PHONY: about
about::
@echo "Makefile"The @echo "Makefile" would be different in each make file to allow identification of the make file. This allows one to run make about and it will show exactly which make files have been included which is really useful when debugging and dealing with multiple included files.
Instead of limiting the amount inclusion to only PRE, MAIN, POST Makefiles, you may also consider something like the following:
INCLUDES ?= makefile-pre makefile-full makefile-post -include $(addprefix $(GENMKFILE_PATH)/,$(INCLUDES))
This will allow one to change the INCLUDES via ENV var and also will not fail Make script if the file does not exist
Still works for me:
https://phabricator.whonix.org/T185#3840
Statements like ifneq ($(GENMKFILE_INCLUDE_FILE_MAIN),0) are pretty much the way to do what we are trying to do with make. Lots of things bug me about make, but it is very powerful.
Alright.
I would think the make file should exit with an error if you are including a file that does not exist, unless it is preceded with a -. Make sure that there are no tabs in the statement. I guess you could also test if the file exists as well like:
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_PRE))) $(error GENMKFILE_INCLUDE_FILE_PRE $(GENMKFILE_INCLUDE_FILE_PRE) does not exist!) endif
Using this now. Works.
One thing I always do is add an about target into all my make files:
.PHONY: about about:: @echo "Makefile"The @echo "Makefile" would be different in each make file to allow identification of the make file. This allows one to run make about and it will show exactly which make files have been included which is really useful when debugging and dealing with multiple included files.
Implemented that. Although I tried it in a generic way, so we can use the exact same file in most places. Please see if you like it.
Instead of limiting the amount inclusion to only PRE, MAIN, POST Makefiles, you may also consider something like the following:
INCLUDES ?= makefile-pre makefile-full makefile-post -include $(addprefix $(GENMKFILE_PATH)/,$(INCLUDES))This will allow one to change the INCLUDES via ENV var and also will not fail Make script if the file does not exist
It's an interesting idea, but then once you set GENMKFILE_PATH you could no longer use the default one. You'd have to leave a symlink in your custom location to the original one if you wanted that.
Alright, that's what I have for now.
#!/usr/bin/make -f
## Copyright (C) 2012 - 2014 Patrick Schleizer <adrelanos@riseup.net>
## See the file COPYING for copying conditions.
## Bootstrapping genmkfile. Using genmkfile to build genmkfile.
## Using genmkfile without having genmkfile available as a build dependency.
.PHONY: about
GENMKFILE_NAME ?= makefile-full
GENMKFILE_BOOTSTRAP ?= ./packages/genmkfile/usr/share/genmkfile
GENMKFILE_PATH ?= $(GENMKFILE_BOOTSTRAP)
GENMKFILE_ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
GENMKFILE_INCLUDE_FILE_MAIN ?= $(GENMKFILE_PATH)/$(GENMKFILE_NAME)
GENMKFILE_CURRENT := $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
export GENMKFILE_NAME
export GENMKFILE_BOOTSTRAP
export GENMKFILE_PATH
export GENMKFILE_ROOT_DIR
export GENMKFILE_INCLUDE_FILE_MAIN
export GENMKFILE_CURRENT
about : ; @echo "GENMKFILE_CURRENT: $(GENMKFILE_CURRENT)"
ifdef GENMKFILE_INCLUDE_FILE_PRE
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_PRE)))
$(error GENMKFILE_INCLUDE_FILE_PRE $(GENMKFILE_INCLUDE_FILE_PRE) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_PRE)
endif
endif
ifneq ($(GENMKFILE_INCLUDE_FILE_MAIN),0)
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_MAIN)))
$(error GENMKFILE_INCLUDE_FILE_MAIN $(GENMKFILE_INCLUDE_FILE_MAIN) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_MAIN)
endif
endif
ifdef GENMKFILE_INCLUDE_FILE_POST
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_POST)))
$(error GENMKFILE_INCLUDE_FILE_POST $(GENMKFILE_INCLUDE_FILE_POST) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_POST)
endif
endifI am not happy about the following syntax.
about : ; @echo "GENMKFILE_CURRENT: $(GENMKFILE_CURRENT)"
But perhaps getting too late here. Following doesn't work.
about: @echo "GENMKFILE_CURRENT: $(GENMKFILE_CURRENT)"
What do you think? Please suggest any required/useful changes.
In case my reply by email does not arrive...
The @echo... is part of a targets rule.. it needs to be indented by one tab, not spaces. One tab will equate to 8 spaces.
about:
<tab here>@echo "GENMKFILE_CURRENT: $(GENMKFILE_CURRENT)"
<blank_line_here>
Alright. So let's take this?
#!/usr/bin/make -f
## Copyright (C) 2012 - 2014 Patrick Schleizer <adrelanos@riseup.net>
## See the file COPYING for copying conditions.
## Bootstrapping genmkfile. Using genmkfile to build genmkfile.
## Using genmkfile without having genmkfile available as a build dependency.
.PHONY: about
GENMKFILE_NAME ?= makefile-full
GENMKFILE_BOOTSTRAP ?= ./packages/genmkfile/usr/share/genmkfile
GENMKFILE_PATH ?= $(GENMKFILE_BOOTSTRAP)
GENMKFILE_ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
GENMKFILE_INCLUDE_FILE_MAIN ?= $(GENMKFILE_PATH)/$(GENMKFILE_NAME)
GENMKFILE_CURRENT := $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
export GENMKFILE_NAME
export GENMKFILE_BOOTSTRAP
export GENMKFILE_PATH
export GENMKFILE_ROOT_DIR
export GENMKFILE_INCLUDE_FILE_MAIN
export GENMKFILE_CURRENT
about:
@echo "GENMKFILE_CURRENT: $(GENMKFILE_CURRENT)"
ifdef GENMKFILE_INCLUDE_FILE_PRE
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_PRE)))
$(error GENMKFILE_INCLUDE_FILE_PRE $(GENMKFILE_INCLUDE_FILE_PRE) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_PRE)
endif
endif
ifneq ($(GENMKFILE_INCLUDE_FILE_MAIN),0)
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_MAIN)))
$(error GENMKFILE_INCLUDE_FILE_MAIN $(GENMKFILE_INCLUDE_FILE_MAIN) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_MAIN)
endif
endif
ifdef GENMKFILE_INCLUDE_FILE_POST
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_POST)))
$(error GENMKFILE_INCLUDE_FILE_POST $(GENMKFILE_INCLUDE_FILE_POST) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_POST)
endif
endifLooks good; I will need to test it to make sure that I can set GENMKFILE_INCLUDE_FILE_MAIN to 0 to skip loading the Makefile.
Good!
One more thing.
Maybe can be avoided by auto detecting the path? It's just 3 different locations we can try. Then the exact same makefile can be used everywhere? No need to have a special bootstrap one? I am unsure about it. When using make from within Whonix/Whonix or genmkfile it should always use the local one?
(With "local" I mean ./packages/genmkfile/usr/share/genmkfile here. Not the system wide installed one.)
I am not exactly sure what you mean.
I do not even know what the current purpose of the make file is. I would personally be using it to build the gateway, build debs, sign tags, update repos... everything. Your makefile can call your scripts as needed.
If I knew what the makefile is currently being used for I could provide better answer. How to you call the make file to begin with? on the command line or with a a script? So if you can explain to me in detail what you want the makefile to do, I can provide some code back :)
It does not look like you need 3 make files that are mostly identical. Usually I would include another makefile named something.conf, which includes and variable overrides or additional targets. It is a makefile, but named with the extension .conf
Or you can just call the master Makefile we have been working on i the ~whonix root directory as follows:
GENMKFILE_BOOTSTRAP=./packages/genmkfile/usr/share/genmkfile make GENMKFILE_BOOTSTRAP=./usr/share/genmkfile make
Best yet is to create some targets and rules.
# make genmkfile
genmkfile: include some_genmkfile_config_file
genmkfile:
@echo 'this is some logic'
if [ some_test == "1" ]; then \
echo 'passed'; \
fi
# To build default workstation and gateway (would read from a .conf file for options)
# make gateway
gateway:
whonix_build $(OPTIONS)
debs:
bash script code to make debs?Just again specifically tested this. Works. You can ignore the main one and source your own make file and targets in pre.
Maybe can be avoided by auto detecting the path? It's just 3 different locations we can try. Then the exact same makefile can be used everywhere? No need to have a special bootstrap one? I am unsure about it. When using make from within Whonix/Whonix or genmkfile it should always use the local one?
(With "local" I mean ./packages/genmkfile/usr/share/genmkfile here. Not the system wide installed one.)
Got this working in this revision.
The purpose of genmkfile is described here:
https://github.com/Whonix/genmkfile/blob/master/debian/control#L18
It's purpose is to avoid having to define "install file x to /usr/bin/x" for each individual file in each individual package. That's something I find awful. Because when you ever rename files in the package, you also have to update the makefile. Too much effort for something as simple as scripts and config files as I find.
With genmkfile each individual package just needs this minimal makefile. It's purpose is to require seldom upgrades. It sources a more comprehensive makefile-full that implements all the real targets such as make install, make installcheck, make uninstall and much more (see make help). Since I find makefiles hard to write (; \ syntax) and debug (no xtrace), the heavy lifting is done by make-helper.bsh. makefile-full and make-helper.bsh which are in the genmkfile package can be often upgraded and all packages benefit from it without requiring to copy/commit a new version.
How to you call the make file to begin with?
For example if troubadour works on whonix-setup-wizard, he can just needs to have genmkfile installed as a build dependency (git clone and "make deb-icup") and can then use in whonix-setup-wizard "make install", "make deb-icup" and others without needing to understand a lot about makefiles and such. It's all nicely abstracted into genmkfile. So can other users/developers easily git clone and play around with a package. When I come up with new hot features such as short commands to quickly bump version number, create and add a package to the repository, I don't have to commit to individual packages such as whonix-setup-wizard.
For Whonix/Whonix I can understand the confusion. It's Makefile is not that useful yet. Some stuff is useful for developers such as "make git-verify". Another plan of mine was to package whole images generated by Whonix/Whonix as deb package. It's not yet in use to build Whonix images. That's only whonix_build.
It does not look like you need 3 make files that are mostly identical.
Yes. Now it autodetects the path. Why are these two exceptions? Well, for genmkfile, I would like to use genmkfile to build genmkfile package. To avoid a circular build dependency (on on itself), I use the same one that comes with genmkfile. And for Whonix/Whonix to avoid having to install genmkfile just to use the makefile.
- To build default workstation and gateway (would read from a .conf file for options)
- make gateway
gateway:
whonix_build $(OPTIONS)debs:
bash script code to make debs?
The makefile for Whonix/Whonix could indeed have additional features that individual packages don't need (make debs, make gateway). Additional features that are only needed for Whonix/Whonix that are not useful for genmkfile. That's another story.
#!/usr/bin/make -f
## Copyright (C) 2012 - 2014 Patrick Schleizer <adrelanos@riseup.net>
## See the file COPYING for copying conditions.
## genmkfile - Makefile - version 1.6
## This is a copy.
## master location:
## https://github.com/Whonix/genmkfile/blob/master/usr/share/genmkfile/Makefile
.PHONY: about
GENMKFILE_NAME ?= makefile-full
GENMKFILE_BOOTSTRAP_ONE ?= ./packages/genmkfile/usr/share/genmkfile/$(GENMKFILE_NAME)
GENMKFILE_BOOTSTRAP_TWO ?= ./usr/share/genmkfile/$(GENMKFILE_NAME)
GENMKFILE_PATH ?= /usr/share/genmkfile
GENMKFILE_ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
GENMKFILE_CURRENT := $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
ifndef GENMKFILE_INCLUDE_FILE_MAIN
ifneq (,$(wildcard $(GENMKFILE_BOOTSTRAP_ONE)))
GENMKFILE_INCLUDE_FILE_MAIN := $(GENMKFILE_BOOTSTRAP_ONE)
else ifneq (,$(wildcard $(GENMKFILE_BOOTSTRAP_TWO)))
GENMKFILE_INCLUDE_FILE_MAIN := $(GENMKFILE_BOOTSTRAP_TWO)
else
GENMKFILE_INCLUDE_FILE_MAIN := $(GENMKFILE_PATH)/$(GENMKFILE_NAME)
endif
endif
export GENMKFILE_NAME
export GENMKFILE_PATH
export GENMKFILE_ROOT_DIR
export GENMKFILE_INCLUDE_FILE_MAIN
export GENMKFILE_CURRENT
about:
@echo "GENMKFILE_CURRENT: $(GENMKFILE_CURRENT)"
ifdef GENMKFILE_INCLUDE_FILE_PRE
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_PRE)))
$(error GENMKFILE_INCLUDE_FILE_PRE $(GENMKFILE_INCLUDE_FILE_PRE) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_PRE)
endif
endif
ifneq ($(GENMKFILE_INCLUDE_FILE_MAIN),0)
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_MAIN)))
$(error GENMKFILE_INCLUDE_FILE_MAIN $(GENMKFILE_INCLUDE_FILE_MAIN) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_MAIN)
endif
endif
ifdef GENMKFILE_INCLUDE_FILE_POST
ifeq (,$(wildcard $(GENMKFILE_INCLUDE_FILE_POST)))
$(error GENMKFILE_INCLUDE_FILE_POST $(GENMKFILE_INCLUDE_FILE_POST) does not exist!)
else
include $(GENMKFILE_INCLUDE_FILE_POST)
endif
endifCommitted:
If there are more points to discuss, please raise them. Otherwise please close this one.