Page MenuHomePhabricator

genmkfile: turn generic makefile into a build dependency to reduce code duplication
Closed, ResolvedPublic

Description

Introduction:

Having the same Makefile and make-helper.bsh in every package is less than ideal.

  • It's a giant code duplication (although packaging-helper-script's function pkg_packaging_files_diff helps to check if they all match).
  • Cumbersome to update.
  • Requires package version bump just because makefile generic was updated even if that update is not required for that package.

Instead a genmkfile package that ships a file /usr/bin/genmkfile could be installed. Instead of make deb-pkg etc., we could then type genmkfile deb-pkg.


Advantages:

  • much less code duplication
  • when grepping the code, fewer results are shown
  • easier to update generic make files
  • in most cases (unrelated generic make file updates), packages would stay the same, would still have an identical hash
  • much less packages have to be updated when updating makefile generic

Disadvantages:

  • new package to be created
  • packages would be no longer "standalone" (they are not anyhow, a lot other build dependencies required to create Debian packages)
  • new build dependency that we would either, and/or:
    • require builders to install before building,
    • or requiring builders to at least adding genmkfile to PATH, [1]
    • or by typing for example /path/to/somewhere/genmkfile deb-pkg instead of make deb-pkg [1]

Why should always all generic makefile be updated?

Otherwise it would be hard to audit ~100 slightly different generic makefiles for non-maliciousness. Since they all match, the following can be used to automate auditing them.

./debug-steps/packaging-helper-scrip pkg_packaging_files_diff

(Or perhaps others are using other custom scripts for such tasks.)


Related:
https://wiki.debian.org/UpstreamGuide


Debian's UpstreamGuide (a pre-condition to get stuff merged into Debian) wants:

  • Out-of-Tree Builds

Maybe it's not all that simple.

  • Major: Debian build tools expect common make targets such as make install and make clean. By having a /usr/bin/genmkfile most likely changes to debian/rules would be required.
  • Minor: having a Makefile around enables make's / bash's competition feature. If we were to expect a genmkfile package by default, we could also ship a bash complementation snippet.

Brainstorming welcome!


[1] Not possible, otherwise would be an "optional build dependency", which would be strange. Either we will be using Build-Depends: genmkfile or not. Making it optional is not possible in any clean way.

Details

Impact
Needs Triage

Event Timeline

Patrick created this task.Mar 5 2015, 7:36 PM
Patrick raised the priority of this task from to Normal.
Patrick updated the task description. (Show Details)
Patrick added a subscriber: Patrick.
Patrick updated the task description. (Show Details)Mar 7 2015, 1:17 AM
Patrick renamed this task from turn generic makefile into a build dependency to reduce code duplication to genmkfile: turn generic makefile into a build dependency to reduce code duplication.Mar 7 2015, 1:32 AM
Patrick added a project: Whonix 10.
Patrick added a subscriber: nrgaway.

Since "update generic makefiles" is on my TODO list, I am wondering if I should solve this one once and for all for Whonix 10.


@nrgaway generally speaking - do you have any opinion on this ticket?

Patrick updated the task description. (Show Details)Mar 7 2015, 1:40 AM
Patrick updated the task description. (Show Details)Mar 7 2015, 1:44 AM
Patrick updated the task description. (Show Details)Mar 7 2015, 1:47 AM
Patrick updated the task description. (Show Details)Mar 7 2015, 1:55 AM
Patrick updated the task description. (Show Details)Mar 7 2015, 1:57 AM
Patrick updated the task description. (Show Details)Mar 13 2015, 4:17 PM
Patrick updated the task description. (Show Details)Apr 1 2015, 7:24 PM
Patrick set Impact to Needs Triage.

Made significant progress. Created a new package genmkfile:
https://github.com/Whonix/genmkfile

90% or how much of Make for packaging scripts can be abstracted in that package.

It will become a build dependency for most of Whonix's packages will depend on. It will need to be build and installed on the build system before building the other packages. (Will be automated by the build script.)

Individual packages (such a whonix-setup-wizard etc. etc.) will then only need to ship a minimal Makefile as their ./Makefile.

#!/usr/bin/make -f

## Copyright (C) 2012 - 2014 Patrick Schleizer <adrelanos@riseup.net>
## See the file COPYING for copying conditions.

## genmkfile - Makefile - version 1.5

## This is a copy.
## master location:
## https://github.com/Whonix/genmkfile/blob/master/usr/share/genmkfile/Makefile

GENMKFILE_PATH ?= /usr/share/genmkfile
GENMKFILE_ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

export GENMKFILE_PATH
export GENMKFILE_ROOT_DIR

include $(GENMKFILE_PATH)/makefile-full

That will be the only - probably unavoidable - code duplication. Hopefully, there will be little need to update that minimal Makefile ever again.

The full Makefile will be included by that specific packages (ex: whonix-setup-wizard) ./Makefile from /usr/share/genmkfile/Makefile. make-helper.bsh will only be in /usr/share/genmkfile/make-helper.bsh. Should the full Makefile or make-helper.bsh be updated, there will hopefully be no need to update specific packages ./Makefiles.

Most likely ready for Whonix 10.

Updated all packages. Example:

TODO:

  • adding the build dependency genmkfile to all packages
  • modifying the build script to automate build and installation fo genmkfile
In T217#3433, @Patrick wrote:

TODO:

  • adding the build dependency genmkfile to all packages

Done.

In T217#3433, @Patrick wrote:

TODO:

  • modifying the build script to automate build and installation fo genmkfile

Done:
https://github.com/Whonix/Whonix/commit/0a05087557d466a3ca0deba5ca4d4dc3165ca7fd

Patrick closed this task as Resolved.Apr 2 2015, 8:59 PM
Patrick claimed this task.