Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support src-one target and backlevel icus #13

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

support src-one target and backlevel icus #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4a7d279
support src-one target and backlevel icus
srl295 Apr 10, 2019
1f31b84
update for review comment
srl295 Apr 11, 2019
399a118
version
srl295 Apr 11, 2019
07772d4
use run --rm so that containers don't dangle
srl295 Apr 11, 2019
2563efd
temp hack to build backlevel
srl295 Jul 19, 2019
6bab9c7
local hacks
srl295 Jul 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,7 @@
*.log
/local
src/tmp
/src/bin/cov-*
/Makefile.local
/pretest-run.pl
/src/bin/local-*
54 changes: 39 additions & 15 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,23 @@ DIRS=tmp src tmp/.ccache dist

ICU_REPO=http://source.icu-project.org/repos/icu/trunk
BINPATH=/src/bin/
# the 'full' list
DISTROS=$(shell cd dockerfiles;ls)
# the 'small' list. The first one is primary.
DISTROS_SMALL=ubuntu fedora
SRC=src/icu
REV:=$(shell (cd $(SRC) && (git describe --tags --exact-match 2>/dev/null || git rev-parse --short HEAD)) || echo 'unknown')
REV:=$(shell (cd $(SRC) >/dev/null 2>/dev/null && (git describe --tags --exact-match 2>/dev/null || git rev-parse --short HEAD)) || echo 'unknown')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of the first >/dev/null here, under what circumstances would cd ever output anything to stdout?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure which circumstances. but it is certainly doing so. might be sh vs bash

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest that you remove the first >/dev/null and in case you later actually encounter a case where it's needed you can then add it with an explanation of the circumstances under which it's needed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was needed on my mac, otherwise I get REV=/some/path 64.2. I will test again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mystery solved: complete user error, https://superuser.com/a/90537/13787

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, I think that unset CDPATH, as recommended there, seems like the most straightforward solution, which leaves no doubt as to what the purpose of it is.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, I think that unset CDPATH, as recommended there, seems like the most straightforward solution

I'm inclined to remove >/dev/null 2>/dev/null everywhere and (possibly) just document to not set CDPATH.

Huh, i found something! See https://bosker.wordpress.com/2012/02/12/bash-scripters-beware-of-the-cdpath/ (kind of a public service announcement):

You can avoid it in some cases by using cd ./foo, which does not consult CDPATH. But this is not a panacea: it can’t easily be used with paths that might be absolute or relative, such as dirname "$0", so I think unsetting CDPATH is still the best way to deal with it.

That actually implies that Q=$(cd ./stuff; echo bar) or P�=$(cd /path/to/stuff; echo baz) might be best - using a non-ambiguous path starting with / or ./.

So setting:

SRC=./src/icu

…at the top of this Makefile actually avoids the CDPATH issue.

#icu4c version
ICU4CVER:=$(shell ./$(BINPATH)/check-icu4c-version.sh $(SRC)/icu4c )
RUNOPTS=--rm
all: dirs src/icu

#src/icu: src
# if [ ! -f src/icu ]; then do cd src ; svn co $(ICU_REPO) icu; done

info:
echo rev $(REV) cver $(ICU4CVER)

dirs:
mkdir -p $(DIRS) tmp/.ccache
chmod a+rwxt tmp/.ccache dist
Expand All @@ -25,55 +33,71 @@ dbuild: docker-compose.yml
build-all: all
for distro in $(DISTROS); do \
echo $$distro ; \
( docker-compose run $$distro bash $(BINPATH)/build.sh || exit 1); \
( docker-compose run $(DOCKERRUN) $$distro bash $(BINPATH)/build.sh || exit 1); \
done
echo all OK
echo all OK $(ICU4CVER)

check-all: all
for distro in $(DISTROS); do \
echo $$distro ; \
rm -f $$distro.fail ; \
( docker-compose run $$distro bash $(BINPATH)/check.sh || (>$$distro.fail; exit 1)) >&1 | tee $$distro.out; \
( docker-compose run $(DOCKERRUN) $$distro bash $(BINPATH)/check.sh || (>$$distro.fail; exit 1)) >&1 | tee $$distro.out; \
done
echo all OK
echo all OK $(ICU4CVER)

check-some: all
for distro in $(DISTROS_SMALL); do \
echo $$distro ; \
rm -f $$distro.fail ; \
( docker-compose run $$distro bash $(BINPATH)/check.sh || (>$$distro.fail; exit 1)) >&1 | tee $$distro.out; \
( docker-compose run $(DOCKERRUN) $$distro bash $(BINPATH)/check.sh || (>$$distro.fail; exit 1)) >&1 | tee $$distro.out; \
done
echo all OK
echo all OK $(ICU4CVER)

dist-all: all
for distro in $(DISTROS); do \
echo $$distro ; \
( docker-compose run $$distro env REV=$(REV) WHAT=$$distro $(BINPATH)/makedist.sh $$distro || exit 1); \
( docker-compose run $(DOCKERRUN) $$distro env REV=$(REV) WHAT=$$distro $(BINPATH)/makedist.sh $$distro || exit 1); \
done
echo all OK
echo all OK $(ICU4CVER)

dist-some: all
for distro in $(DISTROS_SMALL); do \
echo $$distro ; \
( docker-compose run $$distro env REV=$(REV) WHAT=$$distro $(BINPATH)/makedist.sh $$distro || exit 1); \
( docker-compose run $(DOCKERRUN) $$distro env REV=$(REV) WHAT=$$distro $(BINPATH)/makedist.sh $$distro || exit 1); \
done
echo all OK
echo all OK $(ICU4CVER)

dist: sdist dist-some

# just source tarballs. A little more resilient to version changes.
src-some: all
for distro in $(DISTROS_SMALL); do \
echo $$distro ; \
( docker-compose run $(DOCKERRUN) $$distro env REV=$(REV) WHAT=$$distro $(BINPATH)/makesrc.sh $$distro || exit 1); \
done
echo all OK $(ICU4CVER)

# just source tarballs. A little more resilient to version changes.
src-one: all
for distro in $(firstword $(DISTROS_SMALL)); do \
echo $$distro ; \
( docker-compose run $(DOCKERRUN) $$distro env REV=$(REV) WHAT=$$distro $(BINPATH)/makesrc.sh $$distro || exit 1); \
done
echo all OK $(ICU4CVER)

sdist: all
for distro in $(firstword $(DISTROS_SMALL)); do \
echo $$distro ; \
( docker-compose run $$distro env REV=$(REV) WHAT=$$distro $(BINPATH)/makesdoc.sh $$distro || exit 1); \
( docker-compose run $(DOCKERRUN) $$distro env REV=$(REV) WHAT=$$distro $(BINPATH)/makesdoc.sh $$distro || exit 1); \
done
echo all OK
echo all OK $(ICU4CVER)

perf-all: all
for distro in $(DISTROS); do \
echo $$distro ; \
( docker-compose run $$distro bash $(BINPATH)/perf.sh $$distro || exit 1); \
( docker-compose run $(DOCKERRUN) $$distro bash $(BINPATH)/perf.sh $$distro || exit 1); \
done
echo all OK
echo all OK $(ICU4CVER)

.PHONY: all dirs build-all dbuild

Expand Down
4 changes: 4 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@ this will make error messages work
checking for ICU version numbers...
```

- just build tarballs (can work on older releases)

make src-one

## Author

Steven R. Loomis
Expand Down
1 change: 1 addition & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ ubuntu:
- ./src:/src
- ./dist:/dist
- ./tmp/.ccache:/home/build/.ccache
- /Users/srl/src/icu/.git/objects:/Users/srl/src/icu/.git/objects

fedora:
build: ./dockerfiles/fedora
Expand Down
22 changes: 22 additions & 0 deletions local-loopy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/bin/sh

HERE=$(pwd)
ICU=${HERE}/src/icu
MAINTS=$(cd ${ICU} >/dev/null ; git for-each-ref --format='%(refname)' refs/remotes/origin/maint | cut -d/ -f3-)

for maint in $MAINTS;
do
echo $maint
if [ -f $(basename ${maint}).log ];
then
echo "skipping…"
else
cd ${ICU}
git reset --hard HEAD # try to unstick
git clean -d -f -x
git checkout $maint || exit 1
cd ${HERE}
make src-one 2>&1 | tee $(basename ${maint}).log
fi
done

35 changes: 35 additions & 0 deletions local-signoff.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/bin/bash


KEY=9731166CD8E23A83BEE7C6D3ACA5DBE1FD8FABF1
SHAFILE=icu4c-SHASUM512.txt
TOP=$(pwd)
LANG=C
for sub in dist/*;
do
echo $sub
VER=$(basename $sub | cut -d- -f2)
echo $VER
cd ${TOP}/${sub}
if [ -f ${SHAFILE} ];
then
echo skipping due to ${SHAFILE}
else
echo in $VER
echo summing icu4c-$(echo $VER | tr . _)-{src,data,docs}.{tgz,zip}
shasum -a 512 icu4c-$(echo $VER | tr . _)-{src,data,docs}.{tgz,zip} > ${SHAFILE}
echo verifying
shasum -c icu4c-SHASUM512.txt || exit 1

echo signing
for file in icu4c-$(echo $VER | tr . _)-{src,data,docs}.{tgz,zip};
do
if [ -f ${file} ];
then
gpg --detach-sign -a -u ${KEY} ${file} || exit 1
fi
done
gpg --clear-sign -a -u ${KEY} ${SHAFILE}
fi
done

17 changes: 17 additions & 0 deletions src/bin/check-icu4c-version.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/sh
# © 2016 and later: Unicode, Inc. and others.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2016?

# License & terms of use: http://www.unicode.org/copyright.html

if [ $# -ne 0 ];
then
cd $1
shift
fi

if [ ! -f source/common/unicode/uversion.h ];
then
echo "unknown-no-uversion"
exit 1
fi
( grep "^#define U_ICU_VERSION " source/common/unicode/uvernum.h || echo unknown-no-define) | cut -d' ' -f3 | tr -d '"'
exit 0
10 changes: 9 additions & 1 deletion src/bin/check.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,12 @@ then
icu-configure.sh || exit 1
fi

exec make -j${CORES} pcheck
# pcheck was added in 52.1
if [ ${ICU4CMAJ} -gt 51 ];
then
VERB=pcheck
else
VERB=check
fi

exec make -j${CORES} ${VERB}
99 changes: 99 additions & 0 deletions src/bin/dist_icu64.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Copyright (C) 2016 and later: Unicode, Inc. and others.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copyright 2016? Is this file a copy of some existing file? If it is, would it be possible to make git show that in the change history?

The way it is now, it looks as if your wrote a new file from scratch, but put the wrong date on it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it's a copy of dist.mk from ICU trunk. I'll fix the date.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it's a copy of dist.mk from ICU trunk.

Ah, from a file in a different repository? That explains why the diff here shows it as new ... I think it could be valuable if you for those files that are copies listed exact paths of the files they're copies of. That'll make it much easier to track down where things came from if that turns out to be needed in the future.

I'll fix the date.

It looks as if you forgot to actually do that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks as if you forgot to actually do that.

No, as I said above, I wanted finish using this script (getting the maint tarballs out the door) before coming back to this PR.

# License & terms of use: http://www.unicode.org/copyright.html
#******************************************************************************
#
# Copyright (C) 2010-2011, International Business Machines Corporation and others. All Rights Reserved.
#
#******************************************************************************
# This is to be called from ../Makefile.in
#
# This will only work if subversion is installed.
# You must checkout ICU4C at the `/icu` or `/icu/icu4c` level - not just `…/source`
# also note that `make dist` does NOT reflect any local modifications, but pulls from HEAD.

top_builddir = .

include $(top_builddir)/icudefs.mk

ICU_MAJ_VERSION:=$(shell echo $(VERSION) | cut -d. -f1)
DISTY_DIR=dist
DISTY_TMP=dist/tmp
DISTY_ICU=$(DISTY_TMP)/icu
DISTY_DATA=$(DISTY_ICU)/source/data
# The following line controls what is removed in the data/ subdirectory for the source tarball.
DISTY_RMV=brkitr coll curr lang locales mappings rbnf region translit xml zone misc/*.txt misc/*.mk unit
DISTY_RMDIR=$(DISTY_RMV:%=$(DISTY_DATA)/%)
DISTY_IN=$(DISTY_DATA)/in
DOCZIP=icu-docs.zip

ICU4CTOP=$(top_srcdir)/..
GITVER=$(shell (cd $(ICU4CTOP) && (git describe --tags --exact-match 2>/dev/null || git rev-parse --short HEAD)) || echo 'unknown')
DISTY_VER=$(shell echo $(VERSION) | tr '.' '_' )
DISTY_PREFIX=icu4c
DISTY_FILE_DIR=$(shell pwd)/$(DISTY_DIR)
DISTY_FILE_TGZ=$(DISTY_FILE_DIR)/$(DISTY_PREFIX)-$(DISTY_VER)-$(GITVER)-src.tgz
DISTY_FILE_ZIP=$(DISTY_FILE_DIR)/$(DISTY_PREFIX)-$(DISTY_VER)-$(GITVER)-src.zip
DISTY_DOC_ZIP=$(DISTY_FILE_DIR)/$(DISTY_PREFIX)-$(DISTY_VER)-$(GITVER)-docs.zip
DISTY_DATA_ZIP=$(DISTY_FILE_DIR)/$(DISTY_PREFIX)-$(DISTY_VER)-$(GITVER)-data.zip
DISTY_DAT:=$(firstword $(wildcard data/out/tmp/icudt$(SO_TARGET_VERSION_MAJOR)*.dat))

DISTY_FILES_SRC=$(DISTY_FILE_TGZ) $(DISTY_FILE_ZIP)
DISTY_FILES=$(DISTY_FILES_SRC) $(DISTY_DOC_ZIP)
# colon-equals because we watn to run this once!
EXCLUDES_FILE:=$(shell mktemp)

$(DISTY_FILE_DIR):
$(MKINSTALLDIRS) $(DISTY_FILE_DIR)

$(DISTY_TMP):
$(MKINSTALLDIRS) $(DISTY_TMP)

$(DISTY_DOC_ZIP): $(DOCZIP) $(DISTY_FILE_DIR)
cp $(DOCZIP) $(DISTY_DOC_ZIP)
ln -sf $(shell basename $(DISTY_DOC_ZIP)) $(DISTY_FILE_DIR)/icu4c-docs.zip
ln -f $(DISTY_DOC_ZIP) $(DISTY_FILE_DIR)/icu4c-$(DISTY_VER)-docs.zip

$(DISTY_DAT):
echo Missing $@
/bin/false

# make sure we get the non-lgpl docs
$(DOCZIP):
-$(RMV) "$(top_builddir)"/doc
"$(MAKE)" -C . srcdir="$(srcdir)" top_srcdir="$(top_srcdir)" builddir=. $@

$(DISTY_FILE_TGZ) $(DISTY_FILE_ZIP) $(DISTY_DATA_ZIP): $(DISTY_DAT) $(DISTY_TMP)
@echo Export icu4c@$(GITVER) to "$(DISTY_TMP)/icu"
-$(RMV) $(DISTY_FILE) $(DISTY_TMP)
$(MKINSTALLDIRS) $(DISTY_TMP)
( cd $(ICU4CTOP)/.. && git archive --format=tar --prefix=icu/ HEAD:icu4c/ ) | ( cd "$(DISTY_TMP)" && tar xf - )
( cd $(DISTY_TMP)/icu/source ; zip -rlq $(DISTY_DATA_ZIP) data )
$(MKINSTALLDIRS) $(DISTY_IN)
echo DISTY_DAT=$(DISTY_DAT)
cp $(DISTY_DAT) $(DISTY_IN)
$(RMV) $(DISTY_RMDIR)
( cd $(DISTY_TMP)/icu ; python as_is/bomlist.py > as_is/bomlist.txt || rm -f as_is/bomlist.txt )
( cd $(DISTY_TMP) ; tar cfpz $(DISTY_FILE_TGZ) icu )
( cd $(DISTY_TMP) ; zip -rlq $(DISTY_FILE_ZIP) icu )
$(RMV) $(DISTY_TMP)
ln -sf $(shell basename $(DISTY_FILE_ZIP)) $(DISTY_FILE_DIR)/icu4c-src.zip
ln -sf $(shell basename $(DISTY_FILE_TGZ)) $(DISTY_FILE_DIR)/icu4c-src.tgz
ln -sf $(shell basename $(DISTY_DATA_ZIP)) $(DISTY_FILE_DIR)/icu4c-data.zip
ln -f $(DISTY_FILE_ZIP) $(DISTY_FILE_DIR)/icu4c-$(DISTY_VER)-src.zip
ln -f $(DISTY_FILE_TGZ) $(DISTY_FILE_DIR)/icu4c-$(DISTY_VER)-src.tgz
ln -f $(DISTY_DATA_ZIP) $(DISTY_FILE_DIR)/icu4c-$(DISTY_VER)-data.zip
ls -l $(DISTY_FILE_TGZ) $(DISTY_FILE_ZIP) $(DISTY_DATA_ZIP)


dist-local: $(DISTY_FILES)

distcheck: distcheck-tgz

DISTY_CHECK=$(DISTY_TMP)/check

distcheck-tgz: $(DISTY_FILE_TGZ)
@echo Checking $(DISTY_FILE_TGZ)
@-$(RMV) $(DISTY_CHECK)
@$(MKINSTALLDIRS) $(DISTY_CHECK)
@(cd $(DISTY_CHECK) && tar xfpz $(DISTY_FILE_TGZ) && cd icu/source && $(SHELL) ./configure $(DISTCHECK_CONFIG_OPTIONS) && $(MAKE) $(DISTCHECK_MAKE_OPTIONS) && $(MAKE) check $(DISTCHECK_MAKE_OPTIONS) ) && (echo "!!! PASS: $(DISTY_FILE_TGZ)" )

17 changes: 16 additions & 1 deletion src/bin/icu-configure.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,19 @@
# License & terms of use: http://www.unicode.org/copyright.html

. /src/bin/icu-dev.sh
exec ${SHELL} /src/icu/icu4c/source/configure $*
if [ $ICU4CMAJ = 58 ];
then
# workaround for xlocale
CONFIGSTUFF="CPPFLAGS=-DU_USE_STRTOD_L=0"
elif [ $ICU4CMAJ = 59 ];
then
# workaround for xlocale
CONFIGSTUFF="CPPFLAGS=-DU_USE_STRTOD_L=0"
else
CONFIGSTUFF=
fi

echo configure options $CONFIGSTUFF $EXTRACONFIGSTUFF

exec ${SHELL} /src/icu/icu4c/source/configure $CONFIGSTUFF $EXTRACONFIGSTUFF $* || (tail config.log ; exit 1)

2 changes: 2 additions & 0 deletions src/bin/icu-dev.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
# setup some vars
export CORES=$(grep -c ^processor /proc/cpuinfo)
export PATH=/src/bin:${PATH}
export ICU4CVER=$(/src/bin/check-icu4c-version.sh /src/icu/icu4c)
export ICU4CMAJ=$(echo ${ICU4CVER} | cut -d. -f1)

# colored output doesn't work as well from here
export TERM=
Loading