diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..4a629fe
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,7 @@
+# virtual environment folder
+venv*/
+# compiled bytecode
+**/__pycache__/
+**/*.pyc
+# Simulation result folders
+**/batch_202*
\ No newline at end of file
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..e72bfdd
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,674 @@
+ GNU GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc.
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+ Preamble
+
+ The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+ The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works. By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users. We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors. You can apply it to
+your programs, too.
+
+ When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+ To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights. Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+ For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received. You must make sure that they, too, receive
+or can get the source code. And you must show them these terms so they
+know their rights.
+
+ Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+ For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software. For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+ Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so. This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software. The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable. Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products. If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+ Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary. To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+ The precise terms and conditions for copying, distribution and
+modification follow.
+
+ TERMS AND CONDITIONS
+
+ 0. Definitions.
+
+ "This License" refers to version 3 of the GNU General Public License.
+
+ "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+ "The Program" refers to any copyrightable work licensed under this
+License. Each licensee is addressed as "you". "Licensees" and
+"recipients" may be individuals or organizations.
+
+ To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy. The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+ A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+ To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+ To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies. Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+ An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License. If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+ 1. Source Code.
+
+ The "source code" for a work means the preferred form of the work
+for making modifications to it. "Object code" means any non-source
+form of a work.
+
+ A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+ The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form. A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+ The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities. However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work. For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+ The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+ The Corresponding Source for a work in source code form is that
+same work.
+
+ 2. Basic Permissions.
+
+ All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met. This License explicitly affirms your unlimited
+permission to run the unmodified Program. The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work. This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+ You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force. You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright. Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+ Conveying under any other circumstances is permitted solely under
+the conditions stated below. Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+ No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+ When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+ 4. Conveying Verbatim Copies.
+
+ You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+ You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+ 5. Conveying Modified Source Versions.
+
+ You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+ a) The work must carry prominent notices stating that you modified
+ it, and giving a relevant date.
+
+ b) The work must carry prominent notices stating that it is
+ released under this License and any conditions added under section
+ 7. This requirement modifies the requirement in section 4 to
+ "keep intact all notices".
+
+ c) You must license the entire work, as a whole, under this
+ License to anyone who comes into possession of a copy. This
+ License will therefore apply, along with any applicable section 7
+ additional terms, to the whole of the work, and all its parts,
+ regardless of how they are packaged. This License gives no
+ permission to license the work in any other way, but it does not
+ invalidate such permission if you have separately received it.
+
+ d) If the work has interactive user interfaces, each must display
+ Appropriate Legal Notices; however, if the Program has interactive
+ interfaces that do not display Appropriate Legal Notices, your
+ work need not make them do so.
+
+ A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit. Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+ 6. Conveying Non-Source Forms.
+
+ You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+ a) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by the
+ Corresponding Source fixed on a durable physical medium
+ customarily used for software interchange.
+
+ b) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by a
+ written offer, valid for at least three years and valid for as
+ long as you offer spare parts or customer support for that product
+ model, to give anyone who possesses the object code either (1) a
+ copy of the Corresponding Source for all the software in the
+ product that is covered by this License, on a durable physical
+ medium customarily used for software interchange, for a price no
+ more than your reasonable cost of physically performing this
+ conveying of source, or (2) access to copy the
+ Corresponding Source from a network server at no charge.
+
+ c) Convey individual copies of the object code with a copy of the
+ written offer to provide the Corresponding Source. This
+ alternative is allowed only occasionally and noncommercially, and
+ only if you received the object code with such an offer, in accord
+ with subsection 6b.
+
+ d) Convey the object code by offering access from a designated
+ place (gratis or for a charge), and offer equivalent access to the
+ Corresponding Source in the same way through the same place at no
+ further charge. You need not require recipients to copy the
+ Corresponding Source along with the object code. If the place to
+ copy the object code is a network server, the Corresponding Source
+ may be on a different server (operated by you or a third party)
+ that supports equivalent copying facilities, provided you maintain
+ clear directions next to the object code saying where to find the
+ Corresponding Source. Regardless of what server hosts the
+ Corresponding Source, you remain obligated to ensure that it is
+ available for as long as needed to satisfy these requirements.
+
+ e) Convey the object code using peer-to-peer transmission, provided
+ you inform other peers where the object code and Corresponding
+ Source of the work are being offered to the general public at no
+ charge under subsection 6d.
+
+ A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+ A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling. In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage. For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product. A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+ "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source. The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+ If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information. But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+ The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed. Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+ Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+ 7. Additional Terms.
+
+ "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law. If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+ When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it. (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.) You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+ Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+ a) Disclaiming warranty or limiting liability differently from the
+ terms of sections 15 and 16 of this License; or
+
+ b) Requiring preservation of specified reasonable legal notices or
+ author attributions in that material or in the Appropriate Legal
+ Notices displayed by works containing it; or
+
+ c) Prohibiting misrepresentation of the origin of that material, or
+ requiring that modified versions of such material be marked in
+ reasonable ways as different from the original version; or
+
+ d) Limiting the use for publicity purposes of names of licensors or
+ authors of the material; or
+
+ e) Declining to grant rights under trademark law for use of some
+ trade names, trademarks, or service marks; or
+
+ f) Requiring indemnification of licensors and authors of that
+ material by anyone who conveys the material (or modified versions of
+ it) with contractual assumptions of liability to the recipient, for
+ any liability that these contractual assumptions directly impose on
+ those licensors and authors.
+
+ All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10. If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term. If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+ If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+ Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+ 8. Termination.
+
+ You may not propagate or modify a covered work except as expressly
+provided under this License. Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+ However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+ Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+ Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License. If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+ 9. Acceptance Not Required for Having Copies.
+
+ You are not required to accept this License in order to receive or
+run a copy of the Program. Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance. However,
+nothing other than this License grants you permission to propagate or
+modify any covered work. These actions infringe copyright if you do
+not accept this License. Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+ 10. Automatic Licensing of Downstream Recipients.
+
+ Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License. You are not responsible
+for enforcing compliance by third parties with this License.
+
+ An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations. If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+ You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License. For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+ 11. Patents.
+
+ A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based. The
+work thus licensed is called the contributor's "contributor version".
+
+ A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version. For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+ In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement). To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+ If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients. "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+ If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+ A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License. You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+ Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+ 12. No Surrender of Others' Freedom.
+
+ If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all. For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+ 13. Use with the GNU Affero General Public License.
+
+ Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work. The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+ 14. Revised Versions of this License.
+
+ The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time. Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation. If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+ If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+ Later license versions may give you additional or different
+permissions. However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+ 15. Disclaimer of Warranty.
+
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+ 16. Limitation of Liability.
+
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+ 17. Interpretation of Sections 15 and 16.
+
+ If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+ END OF TERMS AND CONDITIONS
+
+ How to Apply These Terms to Your New Programs
+
+ If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+ To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+
+ Copyright (C)
+
+ This program is free software: you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation, either version 3 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program. If not, see .
+
+Also add information on how to contact you by electronic and paper mail.
+
+ If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+ Copyright (C)
+ This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+ This is free software, and you are welcome to redistribute it
+ under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License. Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+ You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+.
+
+ The GNU General Public License does not permit incorporating your program
+into proprietary programs. If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library. If this is what you want to do, use the GNU Lesser General
+Public License instead of this License. But first, please read
+.
\ No newline at end of file
diff --git a/README.md b/README.md
index c725606..3d45ce0 100644
--- a/README.md
+++ b/README.md
@@ -1,2 +1,58 @@
-# OS_ORIGAME
-ORIGAME is a Python-based discrete event modelling and simulation environment. It has a full-featured graphical user interface (GUI) in which users build models and run Monte Carlo simulations.
+# ORIGAME
+Private repository for ORIGAME simulation software.
+
+This repository was created for a Government of Canada contract to update the ORIGAME code base.
+
+(c) Her Majesty the Queen in Right of Canada
+
+## LICENSE
+See LICENSE file.
+
+## RECOMMENDED INSTALLATION
+
+These instructions are for running ORIGAME on python 3.8 and 3.11
+
+1. Install Python 3.8.10 (python-3.8.10-amd64.exe)
+ - https://www.python.org/downloads/release/python-3810/
+
+2. Install Python 3.11.2 (python-3.11.2-amd64.exe)
+ - https://www.python.org/downloads/release/python-3112/
+
+3. Install Visual C++ Redistributable for Visual Studio 2015 (vc_redist.x64.exe)
+ - https://www.microsoft.com/en-ca/download/details.aspx?id=48145
+
+4. Clone or download ORIGAME to a project folder on your system
+
+5. From the project folder, create a virtual environment for ORIGAME for each Python version
+ - e.g. `C:\Python38\python -m venv venv8`
+ - e.g. `C:\Python311\python -m venv venv11`
+
+6. Activate a virtual environment and install dependencies in "requirements.txt". Deactivate the virtual environment
+if not in use.
+ - `venv11\Scripts\activate`
+ - `pip install -r requirements.txt`
+ - `deactivate`
+
+7. Activate the desired virtual environment, and launch ORIGAME GUI.
+ - `venv11\Scripts\activate`
+ - `py .\origame_gui.py`
+
+Visit this [this page](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#activating-a-virtual-environment) for more information about virtual environments.
+
+## DOCUMENTATION
+
+The ORIGAME User Manual and ORIGAME Tutorial documents are located in the /origame/docs folder.
+
+## TESTING
+
+A number of test scenarios and run procedures are provided in the /testing folder.
+
+These test scenarios constitute Government Supplied Material 2 (GSM 2), referred to in the task Statement of Work.
+
+## CONTACT
+
+Stephen Okazawa
+Defence Scientist
+Defence Research and Development Canada
+stephen.okazawa@forces.gc.ca
+
diff --git a/origame/LICENSE.txt b/origame/LICENSE.txt
new file mode 100644
index 0000000..d31a61a
--- /dev/null
+++ b/origame/LICENSE.txt
@@ -0,0 +1,60 @@
+************************
+ORIGAME SOFTWARE LICENSE
+************************
+
+
+Ownership and Administration
+============================
+
+The ORIGAME software and all accompanying documentation and material (hereinafter
+called "the Software") are subject to copyright, with all rights reserved to Her
+Majesty the Queen in Right of Canada.
+
+The Software is administered by Defence Research & Development Canada (DRDC), located at
+60 Moodie Drive, Ottawa, Ontario, K1A 0K2.
+
+
+Approved Users
+==============
+
+The Software may only be used by the following individuals (hereinafter called "Approved Users"):
+
+1. Employees of the Government of Canada.
+
+2. Contractors that are currently engaged in a contract with the Government of
+ Canada and that have a requirement to use the Software in the execution of their work
+ for the Government of Canada.
+
+Individuals other than Approved Users may not use the Software and must destroy any copies of the
+Software in their possession unless they have obtained prior written permission from DRDC to use
+the Software.
+
+
+Approved Use
+============
+
+Users of the Software must adhere to the following restrictions regarding their use
+of the Software (herinafter called "Approved Use"):
+
+1. The Software must only be used to support the execution of work
+ for the Government of Canada.
+
+2. The Software must not be modified without prior written permission from DRDC.
+
+3. The Software must not be provided to others unless the recipients are
+ Approved Users and their use of the Software adheres with the Approved Use.
+
+4. Contractors must further adhere to the terms of their contract with the
+ Government of Canada with respect to their use of the Software.
+
+
+Contact
+=======
+
+ Stephen Okazawa
+ Defence Scientist, DRDC
+ 101 Colonel By Drive, Ottawa, Ontario, K1A 0K2
+ Tel.: (613) 901-9772
+ E-mail: stephen.okazawa@forces.gc.ca
+
+
diff --git a/origame/__init__.py b/origame/__init__.py
new file mode 100644
index 0000000..5c431a9
--- /dev/null
+++ b/origame/__init__.py
@@ -0,0 +1,25 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: This package provides functionality related to running either variant of
+the Origame application.
+
+Version History: See SVN log.
+"""
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5788$"
+
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- PUBLIC API ---------------------------------------------------------------------------------
+# import *public* symbols (classes/functions/constants) from contained modules:
diff --git a/origame/batch_sim/__init__.py b/origame/batch_sim/__init__.py
new file mode 100644
index 0000000..12531e5
--- /dev/null
+++ b/origame/batch_sim/__init__.py
@@ -0,0 +1,28 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: This package provides functionality related to batch simulations in either variant of Origame.
+
+Version History: See SVN log.
+"""
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5788$"
+
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- PUBLIC API ---------------------------------------------------------------------------------
+# import *public* symbols (classes/functions/constants) from contained modules:
+
+from .batch_sim_manager import BatchSimManager, BatchSimSettings, BsmStatesEnum, BatchDoneStatusEnum
+from .batch_sim_manager import MIN_REPLIC_ID, MIN_VARIANT_ID
+from .seed_table import SeedTable
diff --git a/origame/batch_sim/batch_sim_manager.py b/origame/batch_sim/batch_sim_manager.py
new file mode 100644
index 0000000..84c2397
--- /dev/null
+++ b/origame/batch_sim/batch_sim_manager.py
@@ -0,0 +1,1339 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Batch simulation management
+
+The BatchSimManager has a state machine which uses a simple pattern: the BSM has a *state* data member which
+gets initialized to a new instance of one of the state classes every time there is a state transition.
+Each state class knows when to transition out, and what state to transition to, but does not know if the
+target state is allowed. So the target state is first created, and if succesful, the "from" state
+sets the BSM to point to the new state, and gets discarded. Otherwise, the from state remains the current
+state. The state classes only implement the behavior that is supported in the given state; since the BSM
+simply forwards state-dependent calls to the current state object, an exception will get raised if the
+current state does not support the required operation.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import functools
+import logging
+import multiprocessing as mp
+import shutil
+import time
+import json
+from datetime import datetime, timedelta
+from enum import IntEnum
+from pathlib import Path
+from copy import deepcopy
+from textwrap import dedent
+from threading import current_thread
+
+# [2. third-party]
+
+# [3. local]
+from ..core import BridgeEmitter, BridgeSignal, safe_slot, BaseFsmState, IFsmOwner, LogCsvFormatter
+from ..core import internal, override, get_enum_val_name, AppSettings
+from ..core.typing import Any, Either, Optional, List, Tuple, Sequence, Set, Dict, Iterable, Callable, PathType
+from ..core.typing import AnnotationDeclarations
+from ..scenario import ScenarioManager, Scenario, SimSteps
+from ..scenario import create_batch_data_file, get_db_path, BatchDataMgr, DataPathTypesEnum, BATCH_TIMESTAMP_FMT
+from ..scenario.defn_parts import RunRolesEnum
+
+from .bg_replication import ReplicSimState, BatchSetup, ReplicSimConfig, ReplicStatusEnum, ReplicationError
+from .bg_replication import run_bg_replic, get_replic_path, ReplicExitReasonEnum
+from .seed_table import SeedTable, MIN_VARIANT_ID, MIN_REPLIC_ID
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+__all__ = [
+ # public API of module
+ 'BatchSimManager',
+ 'BatchSimSettings',
+ 'BsmStatesEnum'
+ 'BatchDoneStatusEnum',
+ 'get_num_cores_actual',
+]
+
+log = logging.getLogger('system')
+
+
+class Decl(AnnotationDeclarations):
+ BatchSimSettings = 'BatchSimSettings'
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+def get_num_cores_actual(cores_wanted: int, total_num_replics: int) -> int:
+ """
+ Get the actual number of cores that will be used at the beginning of the batch sim run.
+ :param cores_wanted: number of cores the user wants
+ :param total_num_replics: total number of replications that will be run
+ """
+ num_cores_avail = mp.cpu_count()
+ if cores_wanted == 0:
+ # if cores=0, will use all available:
+ return min(num_cores_avail, total_num_replics)
+ else:
+ # if specified, can only use as many as are available, or the total replications, whichever is smaller:
+ return min(num_cores_avail, total_num_replics, cores_wanted)
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+class BatchDoneStatusEnum(IntEnum):
+ not_started, in_progress, paused, aborted, completed = range(5)
+
+
+class BsmStatesEnum(IntEnum):
+ """
+ Enumerate the various states available to the Batch Sim Manager
+ """
+ ready, running, paused, done = range(4)
+
+
+# noinspection PyProtectedMember
+class _BsmStateReady(BaseFsmState):
+ """
+ The only thing that can be done while in READY state is to set the scenario path, load seeds from another
+ file, and start the batch sim.
+
+ plantuml
+ """
+
+ state_id = BsmStatesEnum.ready
+
+ def __init__(self, prev_state: BaseFsmState = None, fsm_owner: IFsmOwner = None):
+ super().__init__(prev_state, fsm_owner=fsm_owner)
+
+ bsm = self._fsm_owner
+ bsm._reset_run_time()
+
+ if prev_state is not None:
+ new_path = prev_state.scen_path_when_ready
+ if new_path is not None:
+ self.set_scen_path(new_path)
+ bsm.signals.sig_replication_done.emit(0, 0)
+
+ def get_completion_status(self) -> BatchDoneStatusEnum:
+ return BatchDoneStatusEnum.not_started
+
+ def start(self):
+ """Start the sim: transitions to RUNNING"""
+ self._set_state(BsmStateClasses.RUNNING)
+
+ def set_scen_path(self, new_path: str):
+ """
+ Set the scenario file path that will be used by each replication of a batch.
+ :param value: scenario file path
+ """
+ log.info("BSM (ready) switching to scenario path '{}'", new_path)
+ self._fsm_owner._scen_path = None if new_path is None else Path(new_path)
+
+ def set_cores_wanted(self, num_cores: int):
+ """
+ It only makes sense to change the cores wanted in READY state. The actual number of cores that will be used
+ depends on how many available, and whether the user wants "all" or a specific #.
+ """
+ # Oliver TODO build 3: add test for this
+ self._fsm_owner._settings.num_cores_wanted = num_cores
+
+
+class CapturesNextReadyScenPath:
+ """
+ The path of a loaded scenario can change at any time, but the path used by a running batch sim cannot
+ change. I.e., the scenario path as seen by the batch sim manager can only change while in the Ready state.
+ So all non-ready BSM states derive from this class to capture any scenario path change, so that it can be
+ "applied" later when BSM returns to its Ready state.
+ """
+
+ def __init__(self):
+ self.__scen_path_when_ready = None
+
+ def set_scen_path(self, new_path: str):
+ """Scenario Path cannot change outside of ready state, but capture so next ready state can use it."""
+ log.info("BSM ({}) save new scenario path '{}' for next Ready state", self._fsm_owner.state_name, new_path)
+ self.__scen_path_when_ready = new_path
+
+ @property
+ def scen_path_when_ready(self) -> Either[str, None]:
+ """If the scenario path changed outside of the READY state, this will contain its value"""
+ return self.__scen_path_when_ready
+
+
+# noinspection PyProtectedMember
+class _BsmStateRunning(BaseFsmState, CapturesNextReadyScenPath):
+ """
+ When entering the RUNNING state from READY, the batch folder is created, and
+ replications are queued and executed in parallel on a specified number of cores.
+ The final exit condition of each replicatoin is recorded by the BatchMonitor. Once
+ all replications have ended, the BSM automatically transitions to DONE.
+
+ NOTE: the init will raise an exception if some preconditions for entering the state
+ are not satisfied, such as the scenario never having been saved (where would the batch sim
+ results go?).
+
+ When entering the RUNNING state from PAUSED, there is no much to do except get a reference
+ to BatchMonitor from the previous state.
+ """
+
+ state_id = BsmStatesEnum.running
+
+ def __init__(self, prev_state: BaseFsmState,
+ fsm_owner: IFsmOwner = None,
+ batch_log_file_handler: logging.Handler = None):
+ """Creates the shared memory manager and pool and initializes replications array and results.
+ :param fsm_owner: owning object of this state machine state
+ :param prev_state: the state object from which BSM transitioning"""
+ BaseFsmState.__init__(self, prev_state, fsm_owner=fsm_owner)
+ CapturesNextReadyScenPath.__init__(self)
+ self.__log_file_handler = batch_log_file_handler
+ self._results_scen_path = None
+
+ # check any pre-conditions else
+ if self._fsm_owner.scen_path is None:
+ raise RuntimeError('Must save scenario before RUNNING')
+
+ settings = self._fsm_owner._settings
+ if settings.auto_seed:
+ assert settings.seed_table is None
+ self.__seed_table = SeedTable(settings.num_variants, settings.num_replics_per_variant)
+ else:
+ self.__seed_table = settings.seed_table
+
+ # ok, init:
+ if prev_state.state_id == BsmStatesEnum.ready:
+ # if previously ready, setup for running
+ self._batch_mon = None
+
+ self._mp_manager = mp.Manager()
+ self._sim_state = ReplicSimState(self._mp_manager)
+ self._worker_pool = None
+
+ self._batch_scen_path = None
+ self._batch_data = None
+
+ elif prev_state.state_id == BsmStatesEnum.paused:
+ # if previously paused, copy state's data
+ self._batch_mon = prev_state._batch_mon
+
+ self._mp_manager = prev_state._mp_manager
+ self._sim_state = prev_state._sim_state
+ self._worker_pool = prev_state._worker_pool
+
+ self._batch_scen_path = prev_state._batch_scen_path
+ self._batch_data = prev_state._batch_data
+
+ else:
+ raise NotImplementedError('Invalid previous state specified for _BsmStateRunning initialization.')
+
+ self._sim_state.paused.value = False
+
+ @override(BaseFsmState)
+ def enter_state(self, prev_state: BaseFsmState):
+ """
+ Entering the RUNNING state must be done separately from object init, because when the state is entered,
+ it queues (if entered from READY) concurrent processes to be run for this batch.
+ Without this separation, replications could complete before the BSM has the new state object as state.
+ """
+ if prev_state.state_id != BsmStatesEnum.ready:
+ # nothing else to do:
+ return
+
+ # so the previous state was ready, setup the batch environment
+ assert self._sim_state.paused.value is False
+
+ bsm = self._fsm_owner
+ settings = bsm._settings
+ total_num_replics = settings.num_variants * settings.num_replics_per_variant
+ num_cores_actual = get_num_cores_actual(settings.num_cores_wanted, total_num_replics)
+
+ assert self.__log_file_handler is None
+ batch_folder = self.__create_batch_folder(settings.num_variants, settings.num_replics_per_variant)
+ self.__save_seed_file(batch_folder)
+ self.__copy_scenario_snapshot(batch_folder)
+
+ # create the monitor of replication processes
+ self._batch_mon = BatchMonitor(bsm, batch_folder, num_cores_actual)
+ create_batch_data_file(batch_folder)
+
+ # create the settings dict that is common to all replications:
+ sim_steps = settings.replic_steps
+ if sim_steps is None:
+ sim_steps = bsm.get_scen_sim_steps()
+ batch_setup = BatchSetup(bsm.scen_path, batch_folder, sim_steps, settings.save_scen_on_exit,
+ **bsm._app_settings)
+
+ # queue a work item for each replication (NxM replications)
+ self._worker_pool = mp.Pool(num_cores_actual, maxtasksperchild=1)
+ for variant_id in range(settings.num_variants):
+ variant_id += MIN_VARIANT_ID
+ for replic_id in range(settings.num_replics_per_variant):
+ replic_id += MIN_REPLIC_ID
+ self._batch_mon.on_replic_queued(variant_id, replic_id)
+ seed = self.__seed_table.get_seed(variant_id, replic_id)
+ replic_sim_config = ReplicSimConfig(variant_id, replic_id, seed)
+ args = (batch_setup, replic_sim_config, self._sim_state,)
+
+ self._worker_pool.apply_async(run_bg_replic, args,
+ callback=self._batch_mon._on_background_replic_done,
+ error_callback=self._batch_mon._on_background_replic_error)
+
+ assert self._batch_mon.get_num_replics_pending() != 0 # no way should get this far with no replics queued!
+ self._worker_pool.close()
+
+ log.info('Queued {} variants, {} replications/variant, to be run among {} cores',
+ settings.num_variants, settings.num_replics_per_variant, num_cores_actual)
+ sim_config = ['{}: {}'.format(key, val) for key, val in bsm._app_settings.items()]
+ log.info('Application config (None value implies default/not-applicable):')
+ for line in sorted(sim_config):
+ log.info(' {}', line)
+
+ @override(BaseFsmState)
+ def exit_state(self, new_state: BaseFsmState):
+ if new_state.state_id == BsmStatesEnum.done:
+ self.__gen_and_save_batch_data()
+
+ def get_completion_status(self) -> BatchDoneStatusEnum:
+ return BatchDoneStatusEnum.in_progress
+
+ def pause(self):
+ """
+ Pause the batch. This affects running replications only: they will each pause. The BSM
+ transitions to PAUSED.
+ """
+ self._set_state(BsmStateClasses.PAUSED, batch_log_file_handler=self.__log_file_handler)
+
+ def stop(self):
+ """Stop the batch. This just sets a flag that each replication not yet run reads. """
+ log.warning('Aborting the batch')
+ self._sim_state.exit.value = True
+ self._worker_pool.terminate()
+ self._set_state(BsmStateClasses.DONE,
+ completion_status=BatchDoneStatusEnum.aborted,
+ batch_log_file_handler=self.__log_file_handler)
+
+ def on_background_replic_done(self):
+ """When a replication is done, check if there are more; if not, transition to DONE state."""
+ if self._batch_mon.get_num_replics_pending() == 0:
+ log.info("Batch sim completed, no more replications left to run")
+ self._set_state(BsmStateClasses.DONE, batch_log_file_handler=self.__log_file_handler)
+
+ def __create_batch_folder(self, num_variants: int, num_replics_per_variant: int) -> Path:
+ """Create a folder to hold all the replication folders, batch log, etc."""
+ fsm_owner = self._fsm_owner
+ assert fsm_owner.scen_path is not None # if None, should have trapped this earlier
+
+ datetime_now = datetime.today().strftime(BATCH_TIMESTAMP_FMT)
+ batch_name = "batch_{}_{}x{}".format(datetime_now, num_variants, num_replics_per_variant)
+ batch_folder = fsm_owner.batch_runs_path / batch_name
+
+ batch_folder.mkdir(parents=True)
+ self.__create_batch_log(batch_folder)
+
+ fsm_owner.signals.sig_batch_folder_changed.emit(str(batch_folder))
+ log.info("Batch sim folder is {}", batch_folder)
+
+ return batch_folder
+
+ def __create_batch_log(self, batch_folder: Path):
+ batch_log = (batch_folder / 'log.csv').absolute()
+ self.__log_file_handler = logging.FileHandler(str(batch_log))
+ self.__log_file_handler.setFormatter(LogCsvFormatter('{asctime},{name},{levelname},"{message}"', style='{'))
+ logging.getLogger('system').addHandler(self.__log_file_handler)
+
+ def __save_seed_file(self, batch_folder: Path):
+ """Save the random-seeds file to the batch folder"""
+ save_path = batch_folder / 'seeds.csv'
+ self.__seed_table.save_as(save_path)
+
+ def __copy_scenario_snapshot(self, batch_folder: Path):
+ """Copy the scenario (last saved version on filesystem) to batch folder"""
+ save_path = batch_folder / Path(self._fsm_owner.scen_path).name
+ log.info("Copying scenario (as last saved) to '{}'", save_path.parent, save_path.name)
+ shutil.copy(str(self._fsm_owner.scen_path), str(save_path))
+ self._batch_scen_path = save_path
+
+ def __gen_and_save_batch_data(self):
+ """Generate batch data per the original batch scenario and save it"""
+ if not BatchDataMgr(self._batch_scen_path.parent).has_data():
+ log.info("No batch data generated by this batch run")
+ return
+
+ scen_mgr = ScenarioManager()
+ try:
+ scen = scen_mgr.load(self._batch_scen_path)
+ except Exception as exc:
+ log.error("Could not load scenario to process batch data: {}", exc)
+ return
+
+ if not scen.sim_controller.has_role_parts(RunRolesEnum.batch):
+ # nothing else to do
+ return
+
+ try:
+ scen.sim_controller.run_parts(RunRolesEnum.batch)
+ except Exception as exc:
+ log.warning('One or more exceptions while trying to run batch-role function parts, see above for details')
+ # TBD FIXME ASAP: remove the dead code below
+ # Reason: need to some time ensure it is not necessary (parts show their errors)
+ # for failed_part, exc_msg in exc.map_part_to_exc_str.items():
+ # log.error(' - {}: {}', failed_part, exc_msg)
+
+ # try to save final state, even if there was an error processing the batch parts:
+ results_scen_path = Path(scen.filepath).with_name('batch_results.orib')
+ try:
+ log.info("Saving batch post-processed state of scenario to {}:", results_scen_path)
+ scen_mgr.save(results_scen_path)
+ except Exception:
+ log.error('Batch post-processed version of scenario could not be saved')
+ else:
+ log.info('Batch post-processed version of scenario saved to {}:', results_scen_path)
+ self._results_scen_path = results_scen_path
+
+
+# noinspection PyProtectedMember
+class _BsmStatePaused(BaseFsmState, CapturesNextReadyScenPath):
+ """
+ Implement the Paused state of the BSM. From this state, the BSM can stop or resume. In order to minimize
+ duplication of behavior, a stop actually causes a transition to RUNNING rather than READY; the RUNNING state
+ knows how to stop (it waits for all replications queued and in-progress to exit). Note that PAUSED must allow
+ for some replications to notify completion since some might have completed their sim loop just before the
+ transition to PAUSED occurred, but before they had completed (hence the on_background_replic_done() needed
+ in PAUSED).
+ """
+
+ state_id = BsmStatesEnum.paused
+
+ def __init__(self, prev_state: BaseFsmState, batch_log_file_handler: logging.Handler, fsm_owner: IFsmOwner = None):
+ BaseFsmState.__init__(self, prev_state, fsm_owner=fsm_owner)
+ CapturesNextReadyScenPath.__init__(self)
+ assert (prev_state.state_id == BsmStatesEnum.running)
+
+ self._batch_mon = prev_state._batch_mon
+
+ self._mp_manager = prev_state._mp_manager
+ self._sim_state = prev_state._sim_state
+ self._worker_pool = prev_state._worker_pool
+
+ self._batch_scen_path = prev_state._batch_scen_path
+ self._batch_data = prev_state._batch_data
+
+ log.info('Pausing running replications')
+ self._sim_state.paused.value = True
+ self.__batch_log_file_handler = batch_log_file_handler
+
+ def get_completion_status(self) -> BatchDoneStatusEnum:
+ return BatchDoneStatusEnum.paused
+
+ def resume(self):
+ """Resume the batch sim; transitions to RUNNING."""
+ log.info('Resuming running replications')
+ self._set_state(BsmStateClasses.RUNNING,
+ batch_log_file_handler=self.__batch_log_file_handler)
+
+ def stop(self):
+ """Flag the replications to exit ASAP, and transition to RUNNING (see class docs for details)."""
+ log.info('Aborting batch sim')
+ self._sim_state.exit.value = True
+ self._worker_pool.terminate()
+ self._set_state(BsmStateClasses.DONE,
+ completion_status=BatchDoneStatusEnum.aborted,
+ batch_log_file_handler=self.__batch_log_file_handler)
+
+ def on_background_replic_done(self):
+ """
+ There is a small chance that a replication could complete after the batch has been paused, if it was
+ already about to exit its sim loop. If there are no more replications left, transition to DONE state.
+ """
+ # Oliver TODO build 3: add test for this: directly call a few times and check via signals
+ if self._batch_mon.get_num_replics_pending() == 0:
+ log.info("Batch sim completed, no more replications left to run")
+ self._set_state(BsmStateClasses.DONE,
+ batch_log_file_handler=self.__batch_log_file_handler)
+
+
+# noinspection PyProtectedMember
+class _BsmStateDone(BaseFsmState, CapturesNextReadyScenPath):
+ """
+ The batch simulation is done, there are no more replications to monitor.
+ The results are kept available, until a transition to READY is requested.
+ """
+
+ state_id = BsmStatesEnum.done
+
+ def __init__(self, prev_state: BaseFsmState,
+ batch_log_file_handler: logging.Handler,
+ completion_status: BatchDoneStatusEnum = BatchDoneStatusEnum.completed,
+ fsm_owner: IFsmOwner = None):
+ BaseFsmState.__init__(self, prev_state, fsm_owner=fsm_owner)
+ CapturesNextReadyScenPath.__init__(self)
+ assert prev_state.state_id in (BsmStatesEnum.running, BsmStatesEnum.paused)
+
+ self._batch_mon = prev_state._batch_mon
+ self._completion_status = completion_status
+ self.__results_scen_path = None
+
+ log.info('Batch {}', completion_status.name)
+ log.info('Summary:')
+ for line in self._batch_mon.get_summary().splitlines():
+ log.info(' {}', line)
+
+ if batch_log_file_handler is not None:
+ logging.getLogger('system').removeHandler(batch_log_file_handler)
+ batch_log_file_handler.close()
+ self.__batch_log_file_path = Path(batch_log_file_handler.baseFilename)
+
+ def enter_state(self, prev_state: BaseFsmState):
+ if prev_state.state_id == BsmStatesEnum.running:
+ self.__results_scen_path = prev_state._results_scen_path
+
+ def get_completion_status(self) -> BatchDoneStatusEnum:
+ return self._completion_status
+
+ def get_batch_log_file_path(self) -> Path:
+ return self.__batch_log_file_path
+
+ def get_batch_results_scen_path(self) -> Path:
+ return self.__results_scen_path
+
+ def new_batch(self):
+ self._set_state(BsmStateClasses.READY)
+
+ def on_background_replic_done(self):
+ """
+ If the batch was stopped before all in-progress replications could end, there is a small chance that
+ multiprocessing could flag them as exited after the DONE state has been entered. However, there is
+ nothing special to do, just need the method to be available.
+ """
+ pass
+
+
+class BatchSimSettings:
+ """
+ Enables saving and loading the configured batch simulation settings with the scenario.
+ """
+
+ # --------------------------- class-wide methods --------------------------------------------
+
+ @staticmethod
+ def load(pathname: PathType) -> Decl.BatchSimSettings:
+ """
+ Load and set the batch simulation settings from the given file. Overrides previous settings if any.
+ :param pathname: path to settings file
+ :returns: The dictionary of batch simulation settings.
+ :raises: ValueError. This error is raised by the JSON interpreter if a parsing error occurs while the file
+ is being loaded.
+ """
+
+ with Path(pathname).open("r") as f:
+ settings = json.load(f)
+
+ # backwards compat:
+ if 'results_root_path' in settings:
+ settings['batch_runs_path'] = settings['results_root_path']
+ del settings['results_root_path']
+
+ seed_list = settings['seed_table']
+ if seed_list is not None:
+ settings['seed_table'] = SeedTable.from_list(seed_list)
+
+ replic_step_settings = settings['replic_steps']
+ if replic_step_settings is not None:
+ settings['replic_steps'] = SimSteps(**replic_step_settings)
+
+ return BatchSimSettings(**settings)
+
+ # --------------------------- instance (self) PUBLIC methods ----------------
+
+ def __init__(self,
+ batch_runs_path: str = None,
+ num_variants: int = 1,
+ num_replics_per_variant: int = 1,
+ num_cores_wanted: int = 0,
+ auto_seed: bool = True,
+ seed_table: SeedTable = None,
+ save_scen_on_exit: bool = True,
+ replic_steps: SimSteps = None,
+ ):
+ """
+ Initialize the batch simulation settings.
+ :param batch_runs_path: The parent folder of all batch run folders.
+ :param num_variants: The number of variants.
+ :param num_replics_per_variant: The number of replications per variant.
+ :param num_cores_wanted: Number of computer cores to use.
+ :param auto_seed: Set True to use automatic seeding.
+ :param seed_table: Instance of the seed table (None is used if auto_seed is True).
+ :param save_scen_on_exit: Set True to save the scenarios.
+ :param replic_steps: Instance of the sim step settings from the scenario's simulation controller.
+ """
+
+ self.batch_runs_path = batch_runs_path
+ self.num_variants = num_variants
+ self.num_replics_per_variant = num_replics_per_variant
+ self.num_cores_wanted = num_cores_wanted
+ self.auto_seed = auto_seed
+ self.seed_table = seed_table
+ self.__check_auto_seeding()
+
+ self.save_scen_on_exit = save_scen_on_exit
+ self.replic_steps = replic_steps
+
+ def save(self, pathname: Path):
+ """
+ Save the batch settings to the given file.
+ :param pathname: The save file path.
+ """
+ settings = self.get_settings_dict()
+ with pathname.open("w") as f:
+ json.dump(settings, f, indent=4, sort_keys=True)
+ log.info('Batch settings saved to {}', pathname)
+
+ def get_use_scen_sim_settings(self) -> bool:
+ """Returns __use_scen_sim_settings boolean based on whether replic_steps is set to None"""
+ return self.replic_steps is None
+
+ def get_settings_dict(self) -> Dict[str, Any]:
+ """Returns a dictionary containing the current batch sim settings"""
+ settings = {
+ 'batch_runs_path': self.batch_runs_path,
+ 'num_variants': self.num_variants,
+ 'num_replics_per_variant': self.num_replics_per_variant,
+ 'num_cores_wanted': self.num_cores_wanted,
+ 'auto_seed': self.auto_seed,
+ 'seed_table': None if self.auto_seed else self.seed_table.get_seeds_list(),
+ 'save_scen_on_exit': self.save_scen_on_exit,
+ 'replic_steps': None if self.replic_steps is None else self.replic_steps.to_json(),
+ }
+
+ return settings
+
+ # --------------------------- instance PUBLIC properties and safe_slots ---------------------
+
+ use_scen_sim_settings = property(get_use_scen_sim_settings)
+
+ # --------------------------- instance _PROTECTED methods ----------------------------
+
+ def __check_auto_seeding(self):
+ """Check that auto-seeding and seed table settings are consistent, fix as necessary."""
+ if self.auto_seed:
+ if self.seed_table is not None:
+ log.warning("Auto-seeding is enabled. Seed Table {} will not be used.", self.seed_table)
+
+ elif self.seed_table is None:
+ self.seed_table = SeedTable(self.num_variants, self.num_replics_per_variant)
+
+ assert self.auto_seed or self.seed_table is not None
+
+
+class BsmStateClasses:
+ READY = _BsmStateReady
+ RUNNING = _BsmStateRunning
+ PAUSED = _BsmStatePaused
+ DONE = _BsmStateDone
+
+
+def ret_val_on_attrib_except(ret_val):
+ """Decorator that will automatically return a specified value if the decorated method raises AttributeError."""
+
+ def decorator(unbound_meth):
+ @functools.wraps(unbound_meth)
+ def wrapper(self, *args, **kwargs):
+ try:
+ return unbound_meth(self, *args, **kwargs)
+ except AttributeError:
+ return ret_val
+
+ return wrapper
+
+ return decorator
+
+
+class BatchSimManager(IFsmOwner):
+ """
+ Manages a batch simulation of scenario replications of scenario variants. Its Signals instance derives from
+ BridgeEmitter so it can emit backend signals when imported in console variant, but emit PyQt signals when
+ imported in the GUI. It forwards several operations to its current
+ state object; if the state does not support the operation, an exception gets raised.
+
+ The manager supports two modes of operation, descirbed in the init.
+ """
+
+ SETTINGS_FILE_EXT = '.bssj'
+ SETTINGS_FILE_NAME = 'batch_sim_settings'
+
+ class Signals(BridgeEmitter):
+ sig_state_changed = BridgeSignal(int) # BsmStatesEnum, but used by BaseFsm which emits as int
+ sig_replication_done = BridgeSignal(int, int) # number of replics done, total number of replics
+ sig_replication_error = BridgeSignal(int, int, str) # num replics done, total num replics, error string
+ sig_num_cores_actual_changed = BridgeSignal(int) # actual number of cores
+ sig_scen_path_changed = BridgeSignal(str) # new path
+ sig_batch_folder_changed = BridgeSignal(str) # new folder for batch
+ # time since last start (stops increasing when Done), number of replics done, number of replics pending,
+ # average ms per replic, estimate to completion (in seconds) from now:
+ sig_time_stats_changed = BridgeSignal(timedelta, int, int, timedelta, timedelta)
+
+ # --------------------------- class-wide methods --------------------------------------------
+
+ @staticmethod
+ def remove_all_batch_folders(path: PathType, max_try_time_sec: int = 10):
+ """
+ Remove all folders in given path.
+ Note: if some folders are currently not removable, will try every 10 ms until max_try_time_sec elapsed.
+ :return: list of folders not deleted within max time
+ """
+ GLOB_PATTERN = "batch_*_*x*"
+ batch_folders = list(Path(path).parent.glob(GLOB_PATTERN))
+ failure_wait_sec = 0.01
+ for batch_folder in batch_folders:
+ fail_removal = True
+ start_time = time.clock()
+ while fail_removal and time.clock() - start_time < max_try_time_sec:
+ try:
+ shutil.rmtree(str(batch_folder))
+ log.debug('Removed {} after {} sec', batch_folder, time.clock() - start_time)
+ fail_removal = False
+ except OSError:
+ time.sleep(failure_wait_sec)
+
+ if fail_removal:
+ log.warning('Could not remove {} in less than {} sec', batch_folder, max_try_time_sec)
+
+ # return list of remaining folders that could not be deleted within max time
+ return list(Path(path).parent.glob(GLOB_PATTERN))
+
+ @classmethod
+ def get_settings_path(cls, scen_path: PathType) -> Path:
+ """Get path to batch sim settings file based on given scenario .ORI(B) file"""
+ return Path(scen_path).with_name(cls.SETTINGS_FILE_NAME).with_suffix(cls.SETTINGS_FILE_EXT)
+
+ # --------------------------- instance (self) PUBLIC methods --------------------------------
+
+ def __init__(self, scenario_manager: ScenarioManager, app_settings: AppSettings = None, bridged_ui: bool = False):
+ """
+ BSM is initialized to a READY state. The presence or absence of a Scenario instance in scenario_manager
+ determines the mode of operation:
+
+ - single scenario mode: this mode is used when the scenario is already avialable from the scenario
+ manager at init time; the manager will not be monitored for scenario replacements, etc, and BSM
+ assumes that app settings will define simulation parameters; the settings from batch sim
+ settings file are loaded; if the sim steps are None but overridden from app settings, they
+ are copied from the scenario's sim controller sim steps and the relevant overrides are applied.
+
+ - multi scenario mode: this mode is used when the scenario is None at init time; then the BSM will
+ monitor the scenario manager for replacement scenario, and will automatically load their settings.
+ """
+ IFsmOwner.__init__(self)
+ self.signals = BatchSimManager.Signals(thread_is_main=True)
+
+ self._app_settings = dict()
+ if app_settings: # use it:
+ self._app_settings.update(
+ save_log=app_settings.save_log,
+ log_deprecated=app_settings.log_deprecated,
+ log_raw_events=app_settings.log_raw_events,
+ fix_linking_on_load=app_settings.fix_linking_on_load,
+
+ bridged_ui=bridged_ui,
+ )
+
+ if hasattr(app_settings, 'loop_log_level') and app_settings.loop_log_level is not None:
+ self._app_settings['loop_log_level'] = app_settings.loop_log_level
+
+ self.__scen_sim_step_settings = None
+ self.__auto_load_settings = True
+ self.__scen_manager = scenario_manager
+ self._scen_path = None
+ self._settings = None
+
+ if scenario_manager.scenario is None:
+ self.__monitor_scen_mgr(scenario_manager)
+ else:
+ # assume this is command line driven and only one scenario will ever be used
+ self.__setup_for_unique_scen(app_settings)
+
+ self._state = _BsmStateReady(fsm_owner=self)
+ assert self._settings.auto_seed is True or self._settings.seed_table is not None
+
+ def get_settings(self, copy: bool = False) -> BatchSimSettings:
+ """
+ By default get a reference to the manager's settings.
+ :param copy: set to True to get a copy of the settings
+ """
+ return deepcopy(self._settings) if copy else self._settings
+
+ def set_settings(self, settings: BatchSimSettings, copy: bool = False):
+ """
+ By default store a reference to the provided settings, which will be used at the next batch run. This
+ method should only be called in the Ready state, otherwise an exception will be raised.
+ :param copy: set to True to make a copy of the settings
+ """
+ self._settings = deepcopy(settings) if copy else settings
+ if self._scen_path is not None:
+ try:
+ self.save_settings()
+ except IOError:
+ log.warning('Failed to save the new settings of BatchSimManager, will try again at next save or set')
+
+ def save_settings(self):
+ """
+ Save the current batch sim manager settings for the loaded scenario. Will fail if no scenario loaded
+ or new scenario never saved.
+ """
+ self._settings.save(self.get_settings_path(self._scen_path))
+
+ def load_settings(self):
+ """
+ Load the batch sim settings for the loaded scenario. Will fail if no scenario loaded
+ or new scenario never saved.
+ """
+ if self._scen_path is None:
+ raise FileNotFoundError('Cannot load batch sim settings (no scenario path yet)')
+
+ self._settings = BatchSimSettings.load(self.get_settings_path(self._scen_path))
+
+ def set_auto_load_settings(self, value: bool = True):
+ """By default, settings are automatically loaded upon scenario change. Set to False to change this."""
+ self.__auto_load_settings = value
+ if self.__auto_load_settings:
+ try:
+ self.load_settings()
+ except FileNotFoundError:
+ log.warning('Auto-loading of settings now True, but no settings file exists (no scenario folder)')
+
+ def get_scen_sim_steps(self, copy: bool = True) -> SimSteps:
+ """
+ Get the scenario simulation step settings, i.e. the simulation steps that are specific to the
+ scenario's simulation controller.
+ :param copy: True if get a copy of settings. Only change
+ :return: the scenario's sim step settings, or a new instance of SimSteps if no scenario loaded/new-not-saved
+ """
+ if self.__scen_sim_step_settings is not None:
+ return SimSteps(**json.loads(self.__scen_sim_step_settings))
+
+ return SimSteps()
+
+ def set_replic_sim_steps(self, **step_settings):
+ """
+ Set the replication sim steps. This causes self.settings.use_scen_sim_settings to become False.
+ :param step_settings: same args as sim_controller.SimSteps.__init__
+ """
+ self._settings.replic_steps = SimSteps(**step_settings)
+ assert self._settings.use_scen_sim_settings is False
+
+ def get_num_cores_wanted(self) -> int:
+ """Get number of cores set for this or next batch"""
+ return self._settings.num_cores_wanted
+
+ def get_num_cores_available(self) -> int:
+ """Get how many cores are available on this machine"""
+ return mp.cpu_count()
+
+ def get_num_variants(self) -> int:
+ """Get number of scenario variants set for this or next batch"""
+ return self._settings.num_variants
+
+ def get_num_replics_per_variant(self) -> int:
+ """Get number of replications per variant set for this or next batch"""
+ return self._settings.num_replics_per_variant
+
+ def get_scen_path(self) -> Path:
+ """Get scenario file path for next batch sim run."""
+ return self._scen_path
+
+ def get_seed_table(self) -> SeedTable:
+ """Get the seed table. If settings.auto_seed is True, returns None"""
+ return self._settings.seed_table
+
+ # In the non-ready states, the following methods will work:
+
+ def is_running(self) -> bool:
+ """Return true if currently in RUNNING state, false otherwise. """
+ return self.is_state(BsmStatesEnum.running)
+
+ def start_sim(self):
+ """Attempt to start a batch sim."""
+ self._state.start()
+
+ def pause_sim(self):
+ """Attempt to pause a batch sim."""
+ # Oliver FIXME ASAP: enable pausing when simualtion does not have events/startup parts
+ self._state.pause()
+
+ def resume_sim(self):
+ """Attempt to resume a batch sim."""
+ self._state.resume()
+
+ def stop_sim(self):
+ """Attempt to stop a batch sim."""
+ self._state.stop()
+
+ def update_sim(self):
+ """
+ Process any state changes that may have resulted from worker threads.
+ Note: this currently doesn't do anything because state is updated by the process thread
+ """
+ pass
+
+ @ret_val_on_attrib_except(0)
+ def get_num_cores_actual(self) -> int:
+ """Get number of cores set for this or next batch"""
+ return self._state._batch_mon.get_num_cores_actual()
+
+ @ret_val_on_attrib_except(0)
+ def get_num_replics_done(self) -> int:
+ """Get number of replications that have started and ended (regardless of success)."""
+ return self._state._batch_mon.get_num_replics_done()
+
+ @ret_val_on_attrib_except(0)
+ def get_num_replics_failed(self) -> int:
+ """Get number of replications that have completed with a failure."""
+ return self._state._batch_mon.get_num_replics_failed()
+
+ @ret_val_on_attrib_except(0)
+ def get_num_variants_done(self) -> int:
+ """Get number of variants that have all their replications completed (regardless of success/fail)."""
+ return self._state._batch_mon.get_num_variants_done()
+
+ @ret_val_on_attrib_except(0)
+ def get_num_variants_failed(self) -> int:
+ """Get number of variants that have all their replications completed but at least one replication failed."""
+ return self._state._batch_mon.get_num_variants_failed()
+
+ @ret_val_on_attrib_except(0)
+ def get_num_replics_in_progress(self) -> int:
+ return self._state._batch_mon.get_num_replics_in_progress()
+
+ @ret_val_on_attrib_except(None)
+ def get_batch_folder(self) -> Optional[Path]:
+ """Get the batch folder of currently running batch sim (or, currently completed batch sim)."""
+ return self._state._batch_mon.get_batch_folder()
+
+ @ret_val_on_attrib_except(None)
+ def get_batch_results_scen_path(self) -> Optional[Path]:
+ """
+ Get the path to the batch results scenario available upon completion of a batch run. Returns None if the
+ file does not exist (because the batch did not complete, or no batch data was generated by any replication).
+ """
+ return self._state.get_batch_results_scen_path()
+
+ @ret_val_on_attrib_except(None)
+ def get_replic_path(self, variant_id: int, replic_id: int) -> Path:
+ """Get the path for the replication from last/current batch"""
+ return self._state._batch_mon.get_replic_path(variant_id, replic_id)
+
+ def wait_till_done(self, max_time_sec=None):
+ """
+ Wait till the BSM is back in ready state, or at most max_time_sec if given. Returns whether the BSM is
+ still in running state. Note: The BSM state may change between the time the return value is created and
+ it is tested by the caller.
+ """
+ # Oliver TODO build 3: add test for this
+ if max_time_sec:
+ start_time = time.clock()
+ while self.is_running() and time.clock() - start_time < max_time_sec:
+ self.update_sim()
+ time.sleep(0.01)
+
+ else:
+ while self.is_running():
+ self.update_sim()
+ time.sleep(0.01)
+
+ return self.is_running()
+
+ def is_state(self, state_id: BsmStatesEnum) -> bool:
+ """Return true if our state object has class state_class"""
+ return state_id == self._state.state_id
+
+ def get_completion_status(self) -> BatchDoneStatusEnum:
+ """Returns the completion status. In ready state, returns not_started"""
+ return self._state.get_completion_status()
+
+ def get_batch_log_file_path(self) -> Path:
+ """When state=DONE, the log file can be obtained"""
+ return self._state.get_batch_log_file_path()
+
+ def get_batch_runs_path(self) -> Path:
+ """
+ Get the path to folder in which a batch folder will be created when a batch is run. If
+ self.settings.batch_runs_path is None then this returns folder containing the scenario file.
+ """
+ if self._settings.batch_runs_path is None:
+ return self.scen_path.parent
+ else:
+ return Path(self._settings.batch_runs_path)
+
+ def new_batch(self):
+ """
+ Abandon any "done" batch and return to Ready state from where a new batch can be configured
+ and started. Can only be called in DONE state.
+ """
+ self._state.new_batch()
+
+ # --------------------------- instance PUBLIC properties ----------------------------
+
+ num_cores_wanted = property(get_num_cores_wanted)
+ num_cores_available = property(get_num_cores_available)
+ num_variants = property(get_num_variants)
+ num_replics_per_variant = property(get_num_replics_per_variant)
+
+ seed_table = property(get_seed_table)
+ scen_path = property(get_scen_path)
+ batch_runs_path = property(get_batch_runs_path)
+ batch_folder = property(get_batch_folder)
+ batch_results_scen_path = property(get_batch_results_scen_path)
+
+ num_cores_actual = property(get_num_cores_actual)
+ num_replics_in_progress = property(get_num_replics_in_progress)
+ num_replics_done = property(get_num_replics_done)
+ num_replics_failed = property(get_num_replics_failed)
+ num_variants_done = property(get_num_variants_done)
+ num_variants_failed = property(get_num_variants_failed)
+
+ settings = property(get_settings)
+
+ # --------------------------- instance __SPECIAL__ method overrides ----------------------------
+ # --------------------------- instance _PROTECTED and _INTERNAL methods ---------------------
+
+ @override(IFsmOwner)
+ def _on_state_changed(self, prev_state: BaseFsmState):
+ """Signal connected slots that our state has changed"""
+ self.signals.sig_state_changed.emit(int(self._state.state_id))
+
+ @internal(BsmStateClasses)
+ def _reset_run_time(self):
+ self.signals.sig_time_stats_changed.emit(timedelta(0), 0, 0, timedelta(0), timedelta(0))
+
+ @internal(BsmStateClasses)
+ def _update_run_time(self, time: timedelta, num_replics_done: int, num_replics_pending: int):
+ avg_ms_per_replic = time / num_replics_done if num_replics_done else 0
+ etc_sec = avg_ms_per_replic * num_replics_pending
+ self.signals.sig_time_stats_changed.emit(time, num_replics_done, num_replics_pending,
+ avg_ms_per_replic, etc_sec)
+
+ # --------------------------- instance _PROTECTED and _INTERNAL properties --------
+ # --------------------------- instance __PRIVATE members-------------------------------------
+
+ def __setup_for_unique_scen(self, app_settings):
+ log.debug("WARNING: using single-scenario mode (scenario available at init time), scen mgr NOT monitored")
+
+ # load the scenario batch settings
+ self.__on_scen_replaced()
+
+ if app_settings:
+ self._settings.num_variants = app_settings.num_variants
+ self._settings.num_replics_per_variant = app_settings.num_replics_per_variant
+ self._settings.num_cores_wanted = app_settings.num_cores
+ self._settings.save_scen_on_exit = app_settings.batch_replic_save
+
+ if app_settings.realtime_scale != 1.0:
+ log.warning('Realtime scale not used during batch, settings ({}) ignored',
+ app_settings.realtime_scale)
+
+ if app_settings.seed_file_path is not None:
+ self._settings.seed_table = SeedTable(
+ self._settings.num_variants,
+ self._settings.num_replics_per_variant,
+ app_settings.seed_file_path)
+ self._settings.seed_table.load()
+ self._settings.auto_seed = False
+
+ if app_settings.max_sim_time_days is not None or app_settings.max_wall_clock_sec is not None:
+ # if using the scen sim steps, get them, because we have to override a portion: when user does
+ # this it is as though they had copied all the other settings over
+ if self._settings.replic_steps is None:
+ assert self.__scen_sim_step_settings is not None
+ self._settings.replic_steps = self.get_scen_sim_steps()
+
+ end_settings = self._settings.replic_steps.end
+ if app_settings.max_sim_time_days is not None:
+ end_settings.max_sim_time_days = app_settings.max_sim_time_days
+ if app_settings.max_wall_clock_sec is not None:
+ end_settings.max_wall_clock_sec = app_settings.max_wall_clock_sec
+
+ def __monitor_scen_mgr(self, scen_mgr: ScenarioManager):
+ """
+ Setup the BSM assuming that the scenario will be delivered later. For now create a default settings
+ obj and connect to scenario manager.
+ """
+ log.debug("WARNING: BSM does not have a scenario yet, assumes will be delivered later")
+ self._settings = BatchSimSettings()
+ assert self._settings.auto_seed is True
+ assert self._settings.seed_table is None
+ assert self._settings.use_scen_sim_settings is True
+
+ scen_mgr_signals = scen_mgr.signals
+ scen_mgr_signals.sig_scenario_replaced.connect(self.__slot_on_scen_replaced)
+ scen_mgr_signals.sig_scenario_saved.connect(self.__slot_on_scen_saved)
+ scen_mgr_signals.sig_scenario_filepath_changed.connect(self.__slot_on_scen_path_changed)
+ # forward signal for scen file path changed:
+ scen_mgr_signals.sig_scenario_filepath_changed.connect(self.signals.sig_scen_path_changed)
+
+ def __on_scen_path_changed(self, filepath: str):
+ log.debug('BSM received signal that scenario file path has changed to {}', filepath)
+ self._state.set_scen_path(filepath or None)
+ if self._scen_path is not None:
+ try:
+ self.load_settings()
+ except FileNotFoundError as exc:
+ log.warning('Could not load scenario {} batch sim settings: {} (like legacy scenario)',
+ self._scen_path, exc)
+
+ def __on_scen_saved(self):
+ if self._scen_path is not None:
+ settings_path = self.get_settings_path(self._scen_path)
+ self._settings.save(settings_path)
+
+ def __on_scen_sim_step_settings_changed(self, json_str: str):
+ self.__scen_sim_step_settings = json_str
+ # scen_sim_step_settings = json.loads(json_str)
+ # self._settings.replic_steps = SimSteps(**scen_sim_step_settings)
+
+ def __on_scen_replaced(self):
+ # Oliver TODO build 3.3: change sig_scen_replaced to carry scenario to slots
+ # Reason: no time before interim release, will need to modify IScenarioMonitor
+ scenario = self.__scen_manager.scenario
+ self._scen_path = scenario.filepath
+ self._settings = BatchSimSettings()
+ if self.__auto_load_settings and self._scen_path is not None:
+ try:
+ self.load_settings()
+ except IOError as exc:
+ log.warning('New scenario loaded, but no batch sim settings file found (likely legacy scenario): {}',
+ exc)
+
+ json_settings = scenario.sim_controller.settings.get_sim_steps(copy=True).to_json()
+ self.__scen_sim_step_settings = json.dumps(json_settings)
+ assert self.__scen_sim_step_settings is not None
+
+ sig_step_settings_changed = scenario.sim_controller.signals.sig_step_settings_changed
+ sig_step_settings_changed.connect(self.__slot_on_scen_sim_step_settings_changed)
+
+ # Oliver FIXME ASAP: define safe_slot slots
+ # Reason: tried this and could not get connections to work, presumably derivations from BridgeEmitter missing
+ __slot_on_scen_path_changed = __on_scen_path_changed
+ __slot_on_scen_saved = __on_scen_saved
+ __slot_on_scen_sim_step_settings_changed = __on_scen_sim_step_settings_changed
+ __slot_on_scen_replaced = __on_scen_replaced
+
+
+class BatchMonitor:
+ """
+ Monitor a batch of replications that will be queued (outside of this class) for multi-processing.
+ The BatchMonitor is useful to keep track of state between the Running, Paused and Done states of
+ the BatchSimManager.
+ """
+
+ def __init__(self, bsm: BatchSimManager, batch_folder: Path, num_cores_start: int):
+ self.__bsm = bsm
+ self.__batch_folder = batch_folder
+ assert batch_folder is not None
+ self.__num_variants = bsm.num_variants
+ self.__num_replics_per_variant = bsm.num_replics_per_variant
+ self.__pool_mutex = mp.RLock() # synchro access to data members accessed by Pool threads AND main thread
+
+ self.__num_cores_actual = num_cores_start
+ self.__replics_in_queue = []
+ self.__replic_results = {} # each replication has a status as result (indicating its completion status)
+
+ self.__start_time = datetime.now()
+ self.__done_time = None
+
+ def on_replic_queued(self, variant_id: int, replic_id: int):
+ """Whenever the BatchSimManager queues a replication for execution, it must notify the monitor."""
+ with self.__pool_mutex:
+ self.__replics_in_queue.append((variant_id, replic_id))
+
+ def get_batch_folder(self) -> Path:
+ """Get the batch folder of currently running batch sim (or, currently completed batch sim)."""
+ return self.__batch_folder
+
+ def get_replic_path(self, variant_id: int, replic_id: int) -> Path:
+ """Get the path for the replication from last/current batch. Raises RuntimeError if batch"""
+ return get_replic_path(self.__batch_folder, variant_id, replic_id)
+
+ def get_num_cores_actual(self) -> int:
+ """
+ Get the number of actual cores in use. This will be less than the original number (num_cores_start)
+ when there are fewer replications left than that number.
+ """
+ with self.__pool_mutex:
+ return self.__num_cores_actual
+
+ def get_num_replics_pending(self) -> int:
+ """Get the number of replications that are not done yet. Note: Some of them might be in progress."""
+ with self.__pool_mutex:
+ return len(self.__replics_in_queue)
+
+ def get_num_replics_in_progress(self) -> int:
+ """
+ Get the number of replication currently executing. This method assumes that it is the number of
+ actual cores in use.
+ """
+ with self.__pool_mutex:
+ return self.__num_cores_actual
+
+ def get_num_replics_done(self) -> int:
+ """
+ Get number of replications that have started and ended, *regardless* of success. So
+ num done - num failed = num completed successfully.
+ """
+ with self.__pool_mutex:
+ num_pending = len(self.__replics_in_queue)
+ num_done = self.__num_variants * self.__num_replics_per_variant - num_pending
+ assert num_done == sum(len(variant_results) for variant_results in self.__replic_results.values())
+ return num_done
+
+ def get_num_replics_failed(self) -> int:
+ """Get number of replications that have failed."""
+ failed = 0
+ with self.__pool_mutex:
+ for variant_replics in self.__replic_results.values():
+ for status in variant_replics.values():
+ if status == ReplicExitReasonEnum.failure:
+ failed += 1
+ return failed
+
+ def get_replics_failed(self) -> List[int]:
+ """Get number of replications that have failed."""
+ failed = []
+ with self.__pool_mutex:
+ for variant_replics in self.__replic_results.values():
+ for replic_id, status in variant_replics.items():
+ if status == ReplicExitReasonEnum.failure:
+ failed.append(replic_id)
+ return sorted(failed)
+
+ def get_num_variants_done(self) -> int:
+ """Get number of variants that have all their replications completed (regardless of success/fail)."""
+ done = 0
+ with self.__pool_mutex:
+ for variant_replics in self.__replic_results.values():
+ if len(variant_replics) == self.__num_replics_per_variant:
+ done += 1
+ return done
+
+ def get_variants_failed(self) -> List[int]:
+ """Get list of variants that have at least one replication failed."""
+ failed_variants = []
+ with self.__pool_mutex:
+ for variant_id, variant_replics in self.__replic_results.items():
+ for status in variant_replics.values():
+ if status == ReplicExitReasonEnum.failure:
+ failed_variants.append(variant_id)
+ break # stop in this variant at first replic failed
+
+ return sorted(failed_variants)
+
+ def get_num_variants_failed(self) -> int:
+ """Get number of variants that have all their replications completed but at least one replication failed."""
+ return len(self.get_variants_failed())
+
+ def get_exec_time(self) -> timedelta:
+ """Return the amount of time used to run a batch simulation"""
+ if self.__done_time is None:
+ return datetime.now() - self.__start_time
+ return self.__done_time - self.__start_time
+
+ def get_summary(self) -> str:
+ num_replics_queued = self.__num_variants * self.__num_replics_per_variant
+ num_variants_failed = self.get_num_variants_failed()
+ num_replics_failed = self.get_num_replics_failed()
+ if num_variants_failed == 0:
+ variants_failed = ''
+ else:
+ variants_failed = '(variant IDs: {})'.format(', '.join(str(id) for id in self.get_variants_failed()))
+ if num_replics_failed == 0:
+ replics_failed = ''
+ else:
+ replics_failed = '(replic IDs: {})'.format(', '.join(str(id) for id in self.get_replics_failed()))
+
+ return dedent("""\
+ Queued at start: {} replications ({} variants)
+ Variants with failures: {} of {} {}
+ Replications failed: {} of {} {}
+ Execution time (days HH:MM:SS.ss): {}
+ """).format(num_replics_queued, self.__num_variants,
+ num_variants_failed, self.get_num_variants_done(), variants_failed,
+ num_replics_failed, self.get_num_replics_done(), replics_failed,
+ self.get_exec_time(),
+ )
+
+ # --------------------------- instance _PROTECTED and _INTERNAL methods ---------------------
+ # --------------------------- instance _PROTECTED and _INTERNAL properties --------
+
+ @internal(_BsmStateRunning)
+ def _on_background_replic_done(self, result: Tuple[int, int, ReplicStatusEnum]):
+ """
+ Called when a replication has completed (returned) successfully
+ :param result: the tuple returned by self._child_replication
+ """
+ variant_id, replic_id, status = result
+ log.info('Got status "{}" for replication ({},{})', get_enum_val_name(status), variant_id, replic_id)
+ with self.__pool_mutex:
+ self.__update_state(variant_id, replic_id, status)
+ # Notify the GUI that replications have been completed.
+ total_replics = self.__num_variants * self.__num_replics_per_variant
+ self.__bsm.signals.sig_replication_done.emit(self.get_num_replics_done(), total_replics)
+
+ @internal(_BsmStateRunning)
+ def _on_background_replic_error(self, exc: ReplicationError):
+ """
+ Called when a replication has raised an exception
+ :param exc: the ReplicationError that was raised
+ """
+ if len(exc.args) < 4:
+ log.error("Unexpected format for ReplicationError! Type {}, args={}: {}", type(exc), exc.args, exc)
+ return
+
+ variant_id, replic_id, err_msg, exc_traceback = exc.args
+ log.error('Replication ({}, {}) raised exception, see its log file for details', variant_id, replic_id)
+ with self.__pool_mutex:
+ status = ReplicExitReasonEnum.failure
+ status.set_exc_traceback(err_msg)
+
+ self.__update_state(variant_id, replic_id, status)
+ total_replics = self.__num_variants * self.__num_replics_per_variant
+ self.__bsm.signals.sig_replication_error.emit(self.get_num_replics_done(), total_replics, err_msg)
+
+ # --------------------------- instance __PRIVATE members-------------------------------------
+
+ def __update_state(self, variant_id: int, replic_id: int, status: ReplicExitReasonEnum):
+ """Update the state of the monitor. Needs to be called whenever the a replication finishes"""
+ self.__done_time = datetime.now()
+
+ variant_results = self.__replic_results.setdefault(variant_id, {})
+ variant_results[replic_id] = status
+
+ self.__replics_in_queue.remove((variant_id, replic_id))
+ self.__update_cores_actual()
+
+ num_pending = len(self.__replics_in_queue)
+ num_done = self.__num_variants * self.__num_replics_per_variant - num_pending
+ self.__bsm._update_run_time(self.get_exec_time(), num_done, num_pending)
+
+ self.__bsm._state.on_background_replic_done()
+
+ def __update_cores_actual(self):
+ """The number of replications left will eventually < num actual cores"""
+ replics_left = len(self.__replics_in_queue)
+ if replics_left < self.__num_cores_actual:
+ self.__num_cores_actual = replics_left
+ if replics_left:
+ log.debug('Batch now using {} cores (1 replication process/core)', self.__num_cores_actual)
diff --git a/origame/batch_sim/bg_replication.py b/origame/batch_sim/bg_replication.py
new file mode 100644
index 0000000..4587e9c
--- /dev/null
+++ b/origame/batch_sim/bg_replication.py
@@ -0,0 +1,630 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Background Replication module
+
+The bg_replication module provides classes that implement the functionality required to run one
+scenario replication. This includes starting, monitoring pause & stop commands from parent
+process (GUI or Console), logging configuration, creation of replication's folder, error handling,
+etc. The module is used primarily by the BatchSimManager class, which uses multiprocessing.Pool to
+start multiple replications in separate child background processes.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import traceback
+import logging
+from enum import IntEnum, unique
+from pathlib import Path
+import multiprocessing as mp
+import sys
+
+# [2. third-party]
+
+# [3. local]
+from ..core import LogManager, log_level_int, log_level_name
+from ..core.utils import ori_profile, ClockTimer
+from ..core.signaling import setup_bridge_for_console
+from ..core.typing import Any, Either, Optional, List, Tuple, Sequence, Set, Dict, Iterable, Callable, PathType
+from ..core.typing import AnnotationDeclarations
+
+from ..scenario import SimController, SimStatesEnum, SimSteps, SimControllerSettings, RunRolePartsError
+from ..scenario import proto_compat_warn, DataPathTypesEnum
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+
+__all__ = [
+ 'run_bg_replic',
+ 'get_replic_path',
+
+ 'ReplicationError',
+ 'ReplicStatusEnum',
+ 'ReplicSimState',
+ 'ReplicSimConfig',
+ 'Replication',
+ 'BatchSetup',
+]
+
+log = logging.getLogger('system')
+
+# set this to True if profile data should be generated for each replication (each one gets its own .pstats file)
+PROFILE_BATCH_REPLICATIONS = False
+
+
+class Decl(AnnotationDeclarations):
+ BatchSetup = 'BatchSetup'
+ ReplicSimConfig = 'ReplicSimConfig'
+ ReplicSimState = 'ReplicSimState'
+ ReplicStatusEnum = 'ReplicStatusEnum'
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+def setup_variant(bridged_ui: bool):
+ """
+ When run from the GUI, the multiprocessing module will load the same modules, same configuration, so the
+ replication ends up with Qt QObject as BridgeEmitter base class to the Scenario signals. Here, we check if
+ this is the case and if so, reload the scenario modules. WARNING: this can cause unexpected behavior because
+ for example enum classes get reloaded; so the same enum member before and after this function is called
+ will have different id() and hence they will not compare equal!
+ """
+ if bridged_ui:
+ log.info('Running batch replication from GUI, patching the signaling module')
+
+ for mod in list(sys.modules.keys()):
+ if mod.startswith('origame.scenario'):
+ del sys.modules[mod]
+
+ setup_bridge_for_console()
+
+ else:
+ # save some time: for console run, no need to re-configure
+ log.info('Running batch replication from Console')
+
+
+def run_bg_replic(batch_setup: Decl.BatchSetup,
+ sim_config: Decl.ReplicSimConfig,
+ shared_sim_state: Decl.ReplicSimState) -> Tuple[int, int, Decl.ReplicStatusEnum]:
+ """
+ Start a replication. This is called by multiprocessing.Pool *in separate process* to start a replication.
+
+ First checks if have an EXIT signal already; only if NOT will a Replication be created and evolved until
+ either STOPPED, no more events, or max sim time reached.
+
+ :param batch_setup: setup parameters common to all replications of a batch
+ :param sim_config: ReplicSimConfig instance containing run parameters specific to sim (random seed etc)
+ :param shared_sim_state: the shared sim state, instance of ReplicSimState
+
+ :returns: (variant_id, replic_id, status), where status is one of ReplicStatusEnum constants
+ :raises ReplicationError: when something went wrong in replication process; this exception got
+ pickled in process and carried over to parent process.
+ """
+
+ # if there has been a batch stop then we want to know before we even create the Replication
+ variant_id, replic_id = sim_config.variant_id, sim_config.replic_id
+
+ log_mgr = LogManager()
+
+ try:
+ shared_sim_state.start(variant_id, replic_id)
+ shared_sim_state.update_exit()
+ if shared_sim_state.need_exit():
+ log.warning("Replication ({},{}) will NOT be created", variant_id, replic_id)
+ return variant_id, replic_id, ReplicExitReasonEnum.stopped
+
+ # ok, replication needed, create its folder; user-facing IDs start at 1 instead of 0
+ replic_path = get_replic_path(batch_setup.batch_folder, variant_id, replic_id)
+ replic_path.mkdir(parents=True)
+ sim_config.replic_path = str(replic_path)
+ if batch_setup.save_log:
+ # and log to log.csv in that folder
+ log_mgr.log_to_file(path=replic_path)
+
+ # start and run it:
+ replication = Replication(batch_setup, sim_config)
+ if PROFILE_BATCH_REPLICATIONS:
+ replication.run = ori_profile(replication.run, batch_setup.scen_path, v=variant_id, r=replic_id)
+ result = replication.run(shared_sim_state)
+
+ return result
+
+ except Exception as exc:
+ log.error('Exception in Replication ({},{}):', variant_id, replic_id)
+ log.exception('Traceback:')
+ # This is tricky because this function gets called in a separate process, so the exception actually
+ # gets pickled by multiprocessing and copied to the host process. It seems that this puts some limitations
+ # on what can be done because after many tries the following was the only reliable way of passing variant
+ # and replication id and relevant info about trackeback
+ exc_tb = traceback.format_exc()
+ raise ReplicationError(variant_id, replic_id, str(exc), exc_tb)
+
+ finally:
+ log_mgr.close()
+
+
+def get_replic_path(batch_folder: str, variant_id: int, replic_id: int) -> Path:
+ """Get the path name to a replication in batch folder with given id's"""
+ return Path(batch_folder) / 'v_{}_r_{}'.format(variant_id, replic_id)
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+class ReplicationError(Exception):
+ """
+ Raised when a replication cannot continue. This may happen during initialization of the replication,
+ during the simulation loop, or (unlikely but possible) during shutdown (when the final scenario state
+ gets saved).
+ """
+
+ def __init__(self, variant_id: int, replic_id: int, message: str, traceback: str):
+ """
+ :param variant_id: id of variant for this replication
+ :param replic_id: id of this replication
+ :param traceback: stack traceback
+ """
+ Exception.__init__(self, variant_id, replic_id, message, traceback)
+
+
+class SimEventExecError(Exception):
+ """
+ Raised when sim controller of a replication has failed processing an event (ie executing the
+ associated IExecutablePart)
+ """
+ pass
+
+
+class ReplicStatusEnum(IntEnum):
+ """
+ Enumeration of the possible status of a scenario replication simulation. Before a Replication instance is
+ created, the replication status is NOT_STARTED. Once it is created, it becomes CREATED, and one the
+ replication starts simulating the scenario, it becomes NOT_DONE. The status will transition to either
+ STOPPED if sim stopped externally, ELAPSED if sim time reached set limit, or NO_MORE_EVENTS if ran out of events.
+ """
+ not_started, initialized, processing_events, exited = range(4)
+
+
+@unique
+class ReplicExitReasonEnum(IntEnum):
+ # normal exits:
+ max_sim_time_elapsed, max_wall_clock_elapsed, no_more_events, paused_by_script = range(4)
+ # abnormal exits (did not end properly):
+ stopped, event_failed, startup_failure, finish_failure = range(20, 24)
+ # general failure:
+ failure = 100
+
+ def set_exc_traceback(self, exc_traceback: str):
+ self.__exc_traceback = exc_traceback
+
+ def get_exc_traceback(self) -> str:
+ return self.__exc_traceback
+
+
+class ReplicSimState:
+ """
+ Represent the simulation state that is shared between background replications and the master process
+ (Origame GUI or Console variant). The master process instantiates only one instance for a batch run,
+ and gives this one instance to each replication. The master does not call any of the methods, but the
+ master writes to self.exit and self.paused when replications should exit or un/pause.
+
+ Each replication runs in a separate process via multiprocessing.Pool and gets its own process-local
+ instance of the ReplicSimState, with initialization state copied across processes by the
+ multiprocessing.Manager, except for self.exit and self.pause which actually reflect the state set
+ by the master process (they are not copies, they dynamically update when master changes).
+
+ Hence each replication calls start() when it starts, and then calls other methods until the replication
+ is eventually done, but it does not change self.exit or self.paused. The
+ multiprocessing module (specifically, its Manager and Pool classes) take care of all the behind-the-scenes
+ communication necessary to transfer self.exit and self.pause across all processes.
+ """
+
+ # calls to update_*() methods are expensive as they communicate with master process; only update every so often:
+ UPDATE_INTERVAL_SEC = 0.1
+
+ def __init__(self, mp_manager: mp.Manager):
+ """
+ :param mp_manager: The multiprocess.Manager instance to use to share exit and pause states between
+ master and replication processes
+ """
+ # shared by all children and parent
+ self.exit = mp_manager.Value('b', False)
+ self.paused = mp_manager.Value('b', False)
+
+ # local to the child receiving self: the instance in master will never see those change; each replication
+ # has its own copy of those data members, but init can only be called in master so can't set here. Use
+ # reasonable defaults:
+ self._need_exit = False
+ self._current_paused = None
+ self._variant_id = None
+ self._replic_id = None
+ self._update_pause_timer = None
+ self._update_exit_timer = None
+
+ def start(self, variant_id: int, replic_id: int):
+ """
+ Signify the start of a replication for the given ID. This is called by the Replication itself,
+ when it starts doing its work, so it is in a separate process!
+ :param variant_id: ID of scenario variant, starts at 0
+ :param replic_id: ID of scenario variant replication, starts at 1
+ """
+ assert variant_id >= 1
+ assert replic_id >= 1
+ self._need_exit = self.exit.value
+ self._current_paused = self.paused.value
+ self._variant_id = variant_id
+ self._replic_id = replic_id
+ self._update_pause_timer = ClockTimer()
+ self._update_exit_timer = ClockTimer()
+
+ def update_paused(self) -> bool:
+ """
+ Update the pause flag based on master setting. This flag state is copied locally (to replication)
+ to ensure it does not change when queried multiple times in one replication step.
+ :return: True if transitioned (previous state different from new), False otherwise
+ """
+ if self._update_pause_timer.total_time_sec < self.UPDATE_INTERVAL_SEC:
+ return False
+
+ self._update_pause_timer.reset()
+ new_paused = self.paused.value
+ if self._current_paused != new_paused:
+ if new_paused:
+ log.info('Replication ({},{}) entering PAUSED state', self._variant_id, self._replic_id)
+ else:
+ log.info('Replication ({},{}) entering RUN state', self._variant_id, self._replic_id)
+ self._current_paused = new_paused
+ return True
+
+ return False
+
+ def need_pause(self) -> bool:
+ """Return True if the Replication should pause. Only call this after update_paused()."""
+ return self._current_paused
+
+ def update_exit(self):
+ """
+ Update the exit flag based on master setting. This flag state is copied locally (to replication)
+ to ensure it does not change when queried multiple times in one replication step.
+ """
+ if self._update_exit_timer.total_time_sec >= self.UPDATE_INTERVAL_SEC:
+ self._update_exit_timer.reset()
+ new_exit = self.exit.value
+ self._need_exit = new_exit
+
+ def need_exit(self) -> bool:
+ """Return True if replication should exit ASAP. Only call this after update_exit()."""
+ return self._need_exit
+
+
+class ReplicSimConfig:
+ """
+ POD structure that aggregates configuration parameters specific to each individual replication
+ of a batch (each replication will have a different instance of this class):
+ its variant and replication ID, its seed, etc.
+ """
+
+ def __init__(self, variant_id: int, replic_id: int, reset_seed: int, replic_path: PathType = None):
+ """
+ :param variant_id: variant id of this replication, starts at 1
+ :param replic_id: replication id of this variant replication, starts at 1
+ :param reset_seed: the seed for the random number generator
+ :param replic_path: path to the replication's folder where log etc stored; if not set now, must be set
+ before the Replication is instantiated
+ """
+ self.variant_id = variant_id
+ self.replic_id = replic_id
+ self.reset_seed = reset_seed
+ self.replic_path = str(replic_path)
+
+
+class BatchSetup:
+ """
+ POD structure that aggregates configuration parameters that are GLOBAL TO THE BATCH, i.e. pertain
+ identically to ALL REPLICATIONS: batch folder, scenario path, whether to log, whether to save on exit,
+ etc.
+ """
+
+ def __init__(self,
+ scen_path: str,
+ batch_folder: str,
+ sim_steps: SimSteps,
+ save_scen_on_exit: bool = True,
+
+ save_log: bool = True,
+ loop_log_level: Either[int, str] = logging.WARNING,
+ log_deprecated: bool = False,
+ log_raw_events: bool = False,
+ fix_linking_on_load: bool = True,
+
+ bridged_ui: bool = False):
+ """
+ :param scen_path: path to scenario file for scenario to run
+ :param batch_folder: the folder in which to save replication folders
+
+ :param save_log: if False, replications will not save their log to a file
+ :param loop_log_level: log level (int or str) for each replication's sim loop
+ :param log_deprecated: if True, use of deprecated functions will be logged
+ :param log_raw_events: if True, simulation events will be logged to a separate file; WARNING:
+ each replication uses the same file, so this option really only makes sense for 1x1 batch!
+ :param fix_linking_on_load: if True, linking will be verified on load and fixed (should only be
+ required for prototype scenarios)
+
+ :param max_sim_time_days: sim time (days) at which replication should exit
+ :param max_wall_clock_sec: real-time (seconds) at which replication should exit
+ :param realtime_scale: scale factor for real-time; if as-fast-as-possible, then None, else must be > 0
+
+ :param save_scen_on_exit: If False, the replication final state (scenario) will not be saved on exit
+ :param bridged_ui: if True, indicates this batch is being run from an application that uses UI bridging; in
+ such case, the replication will re-configure itself without bridging
+ """
+
+ self.scen_path = scen_path
+ self.batch_folder = batch_folder
+ self.sim_steps = sim_steps
+
+ self.save_log = save_log
+ self.loop_log_level = log_level_int(loop_log_level) # always a number
+ self.log_deprecated = log_deprecated
+ self.log_raw_events = log_raw_events
+ self.fix_linking_on_load = fix_linking_on_load
+
+ self.save_scen_on_exit = save_scen_on_exit
+ self.bridged_ui = bridged_ui
+
+
+class SimStartupError(Exception):
+ pass
+
+
+class Replication:
+ """
+ Represent an Origame scenario replication executing the scenario logic. Each replication has its
+ own folder and log file.
+ """
+
+ def __init__(self, batch_config: BatchSetup, sim_config: ReplicSimConfig):
+ """
+ :param batch_config: batch-level configuration parameters (scenario path, batch folder, etc)
+ :param sim_config: replication-specific settings (variant and replication ID, replication folder etc)
+ """
+
+ variant_id = sim_config.variant_id
+ replic_id = sim_config.replic_id
+
+ setup_variant(batch_config.bridged_ui)
+
+ # Signaling-related functionality must be imported here due to how multiprocessing imports modules in processes;
+ # for example, if the batch is started from GUI, we still want Replication's ScenarioManager to use
+ # BackendEmitter not BridgeEmitter
+ from ..core.signaling import BackendEmitter
+ from ..scenario import ScenarioManager, MIN_REPLIC_ID, MIN_VARIANT_ID
+ assert issubclass(ScenarioManager.Signals, BackendEmitter)
+
+ # Each replication has its own folder
+ # create folder for this replication, inside batch folder:
+ log.info('Creating Replication ({},{})', variant_id, replic_id)
+
+ if variant_id < MIN_VARIANT_ID:
+ raise ValueError("invalid variant id", variant_id)
+ if replic_id < MIN_REPLIC_ID:
+ raise ValueError("invalid replication id", replic_id)
+
+ self.__v_id = variant_id
+ self.__r_id = replic_id
+ self.__replic_folder = sim_config.replic_path
+ self.__save_scen_on_exit = batch_config.save_scen_on_exit
+ self.__sim_loop_log_level = batch_config.loop_log_level
+
+ self.__replic_status = ReplicStatusEnum.initialized
+
+ # load scenario
+ self.__scenario_mgr = ScenarioManager()
+ # assert Path(replic_folder).parent.parent == Path(scen_path).parent
+ self.__scenario_mgr.config_logging(batch_config)
+ self.__scenario_mgr.set_future_anim_mode_constness(False)
+ scen = self.__scenario_mgr.load(batch_config.scen_path)
+ assert scen.scenario_def.root_actor.anim_mode is False
+ # WARNING: due to setup_variant() re-importing modules, we cannot provide the file_type here, it will not
+ # compare equal. Instead we let batch_data module infer it.
+ scen.shared_state.batch_data_mgr.set_data_path(batch_config.batch_folder)
+ # scen.shared_state.batch_data_mgr.set_data_path(batch_config.batch_folder,
+ # file_type=DataPathTypesEnum.batch_folder) # FAILS, see above
+
+ # config sim controller of that scenario
+ sim_settings = SimControllerSettings(variant_id=variant_id,
+ replic_id=replic_id,
+ auto_seed=False, # batch manager always picks seed for replication
+ reset_seed=sim_config.reset_seed,
+ sim_steps=batch_config.sim_steps)
+ self.__sim_controller = self.__scenario_mgr.scenario.sim_controller
+ self.__sim_controller.replic_folder = sim_config.replic_path
+ self.__sim_controller.set_settings(sim_settings)
+ assert self.__sim_controller.get_anim_while_run_dyn_setting() is True
+ assert self.__sim_controller.is_animated is False
+
+ @property
+ def status(self) -> ReplicStatusEnum:
+ """Obtain run status of the replication."""
+ return self.__replic_status
+
+ @property
+ def replic_folder(self):
+ """Get the folder for this replication"""
+ return self.__replic_folder
+
+ @property
+ def sim_controller(self) -> SimController:
+ """Get the sim controller for this replication"""
+ return self.__sim_controller
+
+ @property
+ def variant_id(self) -> int:
+ """Get the sim controller for this replication"""
+ return self.__v_id
+
+ @property
+ def replic_id(self) -> int:
+ """Get the sim controller for this replication"""
+ return self.__r_id
+
+ def run(self, shared_sim_state: ReplicSimState) -> Tuple[int, int, ReplicStatusEnum]:
+ """
+ Start a replication, with given shared sim state. Will be evolved in a loop until either STOPPED
+ (via shared sim state), no more events, or max sim time reached.
+
+ :param shared_sim_state: the shared sim state, instance of ReplicSimState
+ :returns: (variant_id, replic_id, status), where status is one of ReplicStatusEnum constants
+ :raises ReplicationError: when something went wrong in replication process; this exception got
+ pickled in process and carried over to parent process.
+ """
+ shared_sim_state.update_exit()
+ if shared_sim_state.need_exit():
+ replic_exit_reason = ReplicExitReasonEnum.stopped
+ self.__replic_status = ReplicStatusEnum.exited
+ log.warning('Replication ({},{}) run stopped before start (STOPPED)', self.__v_id, self.__r_id)
+ return self.__v_id, self.__r_id, replic_exit_reason
+
+ try:
+ self.__startup(shared_sim_state)
+ replic_exit_reason = self.__loop_till_stopped(shared_sim_state)
+
+ # process exit condition:
+ if replic_exit_reason == ReplicExitReasonEnum.stopped:
+ assert shared_sim_state.need_exit()
+ log.warning('Unsuccessful completion for Replication ({},{}) (STOPPED)', self.__v_id, self.__r_id)
+
+ elif replic_exit_reason == ReplicExitReasonEnum.event_failed:
+ last_step_error_info = self.__sim_controller.last_step_error_info
+ assert self.__sim_controller.last_step_was_error
+ error_msg = last_step_error_info.msg
+ log.error(error_msg)
+ log.error('Unsuccessful completion for Replication ({},{}) (FAILED event)', self.__v_id, self.__r_id)
+ raise SimEventExecError(error_msg)
+
+ else:
+ assert not self.__sim_controller.last_step_was_error
+ log.info('Successful completion for Replication ({},{})', self.__v_id, self.__r_id)
+
+ # done:
+ return self.__v_id, self.__r_id, replic_exit_reason
+
+ finally:
+ self.__replic_status = ReplicStatusEnum.exited
+ # After scenario saved-as, scenario path will be in the replication folder, and on scenario shutdown,
+ # batch replication data automatically gets saved if there is any. *So* we have to save the batch
+ # replication data *first* AND clear it, so it doesn't get saved in the wrong place on scenario shutdown.
+ self.__scenario_mgr.scenario.save_batch_replic_data(clear_after=True)
+ # regardless of success, attempt to save scenario in case final state useful for debugging
+ if self.__save_scen_on_exit:
+ self.__scenario_mgr.save(Path(self.__replic_folder, 'final_scenario.ori'))
+
+ # --------------------------- instance _PROTECTED and _INTERNAL methods ---------------------
+ # --------------------------- instance _PROTECTED properties and safe slots -----------------
+ # --------------------------- instance __PRIVATE members-------------------------------------
+
+ def __startup(self, shared_sim_state: ReplicSimState):
+ try:
+ log.info('Starting Replication ({},{})', self.__v_id, self.__r_id)
+ self.__replic_status = ReplicStatusEnum.processing_events
+ # even if this replic has been paused already, we need to start the sim then check for pause, else too
+ # much logic repeated:
+ self.__sim_controller.sim_run()
+ # now check if should be in pause state before entering the loop :
+ shared_sim_state.update_paused()
+ if shared_sim_state.need_pause():
+ self.__sim_controller.sim_pause()
+
+ except Exception as exc:
+ # log.error('Failed startup: {}', exc)
+ raise SimStartupError('Failed startup: {}'.format(exc))
+
+ def __loop_till_stopped(self, shared_sim_state: ReplicSimState) -> ReplicExitReasonEnum:
+ prev_level = log.getEffectiveLevel()
+ if prev_level != self.__sim_loop_log_level:
+ log.warning("Changing log level to {} for sim loop", log_level_name(self.__sim_loop_log_level))
+ log.setLevel(self.__sim_loop_log_level)
+
+ try:
+ shared_sim_state.update_exit()
+ replic_exit_reason = None
+ while replic_exit_reason is None:
+ if shared_sim_state.update_paused():
+ if shared_sim_state.need_pause():
+ self.__sim_controller.sim_pause()
+ else:
+ self.__sim_controller.sim_resume()
+
+ replic_exit_reason = self.__step(shared_sim_state)
+
+ return replic_exit_reason
+
+ except RunRolePartsError as exc:
+ log.error(str(exc))
+ raise
+
+ finally:
+ if prev_level != self.__sim_loop_log_level:
+ log.warning("Sim loop done, restoring log level to {}", logging.getLevelName(prev_level))
+ log.setLevel(prev_level)
+
+ def __scen_transitioned_to_paused(self, before_state: SimStatesEnum):
+ return before_state != SimStatesEnum.paused and self.__sim_controller.state_id == SimStatesEnum.paused
+
+ def __step(self, shared_sim_state: ReplicSimState) -> ReplicExitReasonEnum:
+ """
+ Execute one step of evolution of the scenario replication. This steps the simulation engine.
+ """
+ assert self.__replic_status == ReplicStatusEnum.processing_events
+ replic_exit_reason = None
+
+ sim_con_state_before = self.__sim_controller.state_id
+ self.__sim_controller.sim_update()
+
+ if self.__sim_controller.last_step_was_error:
+ log.error('Replication {},{} failed to process an event', self.__v_id, self.__r_id)
+ replic_exit_reason = ReplicExitReasonEnum.event_failed
+
+ elif self.__sim_controller.max_sim_time_elapsed:
+ # assert self.__sim_controller.is_state(SimStatesEnum.paused)
+ log.warning('Replication {},{} reached max sim date-time {}',
+ self.__v_id, self.__r_id, self.__sim_controller.sim_time_days)
+ replic_exit_reason = ReplicExitReasonEnum.max_sim_time_elapsed
+
+ elif self.__sim_controller.max_wall_clock_elapsed:
+ log.warning('Replication {},{} reached max real time {} (excluding pause times)',
+ self.__v_id, self.__r_id, self.__sim_controller.realtime_sec)
+ replic_exit_reason = ReplicExitReasonEnum.max_wall_clock_elapsed
+
+ elif self.__sim_controller.num_events == 0:
+ # WARN level otherwise risk not seeing since part of event loop
+ log.warning('Replication {},{} consumed all events', self.__v_id, self.__r_id)
+ replic_exit_reason = ReplicExitReasonEnum.no_more_events
+
+ elif self.__scen_transitioned_to_paused(sim_con_state_before):
+ log.warning('Replication {},{} paused by scenario, cannot continue', self.__v_id, self.__r_id)
+ replic_exit_reason = ReplicExitReasonEnum.paused_by_script
+
+ else:
+ # might have been stopped externally:
+ shared_sim_state.update_exit()
+ if shared_sim_state.need_exit():
+ self.__sim_controller.sim_pause()
+ replic_exit_reason = ReplicExitReasonEnum.stopped
+
+ return replic_exit_reason
diff --git a/origame/batch_sim/seed_table.py b/origame/batch_sim/seed_table.py
new file mode 100644
index 0000000..d9c1172
--- /dev/null
+++ b/origame/batch_sim/seed_table.py
@@ -0,0 +1,342 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: This module exposes functions that can be used to read/write to a seed file.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import csv
+import logging
+from random import Random
+from pathlib import Path
+
+# [2. third-party]
+
+# [3. local]
+from ..core.typing import AnnotationDeclarations
+from ..core.typing import Any, Either, Optional, Callable, PathType, TextIO, BinaryIO
+from ..core.typing import List, Tuple, Sequence, Set, Dict, Iterable, Stream
+from ..scenario.sim_controller import MIN_RAND_SEED, MAX_RAND_SEED, MIN_VARIANT_ID, MIN_REPLIC_ID
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+
+__all__ = [
+ 'SeedFileFormatError',
+ 'SeedFileIncompleteError',
+ 'SeedFileInvalidInformationError',
+ 'SeedTable',
+ 'MIN_VARIANT_ID',
+ 'MIN_REPLIC_ID',
+]
+
+log = logging.getLogger('system')
+
+LINE_OFFSET = 1 # Offset to account for CSV file header line
+
+
+class Decl(AnnotationDeclarations):
+ SeedTable = 'SeedTable'
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+class SeedFileFormatError(Exception):
+ """Raised when a seed file is being read has wrong format"""
+
+ def __init__(self, msg):
+ super().__init__(msg)
+
+
+class SeedFileIncompleteError(Exception):
+ def __init__(self, variant_id, num_missing):
+ super().__init__("Missing seeds for desired seed table", variant_id, num_missing)
+
+
+class SeedFileInvalidInformationError(Exception):
+ """Raised when the format is correct but the info is invalid, such as a value not being within allowed range. """
+
+ def __init__(self, msg):
+ super().__init__(msg)
+
+
+class SeedTable:
+ """
+ Represent the random seeds to be used for a batch simulation of N variants by M replications per variant.
+ The table can be saved to file in CSV format, and loaded from CSV.
+
+ The CSV file must be in the format 'variant, replication, seed' with a header 'var,rep,seed'.
+ An example of a valid csv file that can be loaded by this class is:
+
+ var,rep,seed
+ 1,1,454
+ 1,2,399
+ 2,1,09923
+ 2,2,876
+ 3,1,56373
+ 3,2,09955
+ """
+
+ # --------------------------- class-wide data and signals -----------------------------------
+
+ SeedList = List[Tuple[int, int, float]]
+
+ VARIANT_COL = 0
+ REPLIC_COL = 1
+ SEED_COL = 2
+
+ _csv_header = ('var', 'rep', 'seed')
+
+ # --------------------------- class-wide methods --------------------------------------------
+
+ @staticmethod
+ def from_list(seed_list: SeedList) -> Decl.SeedTable:
+ """Converts a list of seeds to a SeedTable object"""
+
+ num_variants = seed_list[-1][0]
+ num_replics_per_variant = seed_list[-1][1]
+ seed_table = SeedTable(num_variants, num_replics_per_variant)
+
+ for variant_id, replic_id, seed in seed_list:
+ seed_table.set_seed(variant_id, replic_id, seed)
+
+ return seed_table
+
+ @staticmethod
+ def get_csv_file_name(filename: PathType):
+ path = Path(filename)
+ if not path.suffix:
+ path = path.with_suffix(".csv")
+
+ if path.suffix != ".csv":
+ raise ValueError("Filename '{}' has invalid suffix, must be .csv".format(filename))
+
+ return path
+
+ # --------------------------- instance (self) PUBLIC methods --------------------------------
+
+ def __init__(self, num_variants: int, num_replics_per_variant: int, csv_path: PathType = None):
+ """
+ If no path given, then random numbers will be generated for given number of variants and replics per variant.
+ Otherwise, the file must contain random seeds for the specified number of variants and replics per variant.
+
+ :param num_variants: The number of variants expected in the seed file
+ :param num_replics_per_variant: The number of replications per num_variants expected in the seed file
+ :param csv_path: If given, file to load seeds from.
+ """
+
+ self._num_variants = num_variants
+ self._replics_per_variant = num_replics_per_variant
+ self._seeds = self._generate_seeds()
+ self._csv_path = None
+ if csv_path:
+ self._csv_path = self.get_csv_file_name(csv_path)
+
+ def get_seeds_list_iter(self) -> Iterable[Tuple[int, int, int]]:
+ """
+ Get a copy of the 2d array (list of list) representation of a seed file.
+ """
+ for variant_id, variant_seeds in enumerate(self._seeds):
+ if variant_id >= MIN_VARIANT_ID:
+ for replic_id, seed in enumerate(variant_seeds):
+ if replic_id >= MIN_REPLIC_ID:
+ yield variant_id, replic_id, seed
+
+ def get_seeds_list(self):
+ return list(self.get_seeds_list_iter())
+
+ def set_seed(self, variant_id: int, replic_id: int, seed: int):
+ """
+ Set the seed for given variant and replication ID to given seed.
+ s from a 2d array (list of list) object. The seeds array is copied entirely, even if it is
+ larger than necessary. Array must have at least num_variants x num_replics_per_variant seeds (given at
+ construction time).
+ :param array2d: the array from which to get the random number generator seeds
+ :raises: ValueError, if insufficient seeds in array for number of variants and replications configured
+ """
+ self._check_data_valid(variant_id, replic_id, seed)
+ try:
+ self._seeds[variant_id][replic_id] = seed
+ except IndexError:
+ raise ValueError("IDs must be >= 1, max variant_id is {}, max replic_id is {}"
+ .format(self._num_variants, self._replics_per_variant))
+
+ def get_num_variants(self) -> int:
+ """Number of variants supported by this seed table"""
+ return self._num_variants
+
+ def get_num_replics_per_variant(self) -> int:
+ """Number of replications per variant supported by this seed table"""
+ return self._replics_per_variant
+
+ def get_seed(self, variant_id: int, replic_id: int) -> int:
+ """
+ Get the random seed for given variant and replication IDs. IDs must be in range of num variants and num
+ replications per variant.
+ """
+ return self._seeds[variant_id][replic_id]
+
+ def copy(self, rhs: Decl.SeedTable):
+ """
+ Copies the seeds from given table into self. Only uses the portion of rhs that fits in self (if rhs larger
+ than self), or (if rhs smaller than self) only overwrites portion of self covered by rhs.
+ """
+ num_variants = min(rhs.num_variants, self._num_variants)
+ num_replics_per_variant = min(rhs.num_replics_per_variant, self._replics_per_variant)
+ for v_id in range(1, num_variants + 1):
+ for r_id in range(1, num_replics_per_variant + 1):
+ self._seeds[v_id][r_id] = rhs.get_seed(v_id, r_id)
+
+ def load(self):
+ """
+ Method used to load a comma delimited CSV file. Each row of the CSV
+ file is a list. Each list contains a variant, replication and seed information.
+
+ :raises: SeedFileFormatError: Raised when number of rows does not match expected number (where expected
+ number is var*rep. Can also be raised if seed file contains missing information (ie within a row).
+ :raises: SeedFileInvalidInformationError: Raised when invalid data is found within the seed file
+ (ie string values instead of numbers). Can also be raised if there are duplicates of any
+ variation-replication pair.
+ :raises: RuntimeError: if csv path to file not specified
+ """
+ if self._csv_path is None:
+ raise ValueError("Must set seed file path before load()")
+
+ # create temporary empty seeds array with sentinel_value values
+ sentinel_value = 0
+ new_seeds = [[sentinel_value] * (self._replics_per_variant + MIN_REPLIC_ID)
+ for _ in range(self._num_variants + MIN_VARIANT_ID)]
+
+ log.debug('Loading seeds from {}', self._csv_path)
+ with self._csv_path.open('r') as file:
+ csv_reader = csv.reader(file, delimiter=",")
+ for row_index, row in enumerate(csv_reader):
+ line_num = row_index + LINE_OFFSET
+ try:
+ variant = int(row[self.VARIANT_COL])
+ replic = int(row[self.REPLIC_COL])
+ seed = int(row[self.SEED_COL])
+ except ValueError:
+ if row_index != 0 or tuple(row) != self._csv_header:
+ msg = "Line {}: some values are not numbers".format(line_num)
+ raise SeedFileInvalidInformationError(msg)
+ else:
+ continue # skip this line
+
+ try:
+ self._check_data_valid(variant, replic, seed)
+ except ValueError as exc:
+ raise SeedFileInvalidInformationError("Line #{}: {}".format(line_num, exc))
+
+ # if the var and rep ID are in range, accept; else, ignore it, so bigger tables can be used
+ if (variant <= self._num_variants) and (replic <= self._replics_per_variant):
+ value = new_seeds[variant][replic]
+ if value != 0:
+ msg = "Line #{}: seed already defined earlier in file".format(line_num)
+ raise SeedFileInvalidInformationError(msg)
+ new_seeds[variant][replic] = seed
+
+ # check that there are no sentinels left except in first row and col, as this would indicate that
+ # there were missing seeds
+ for variant_id, var_seeds in enumerate(new_seeds):
+ if variant_id >= MIN_VARIANT_ID:
+ num_zeros = var_seeds.count(sentinel_value)
+ if num_zeros > MIN_REPLIC_ID:
+ raise SeedFileIncompleteError(variant_id, num_zeros)
+
+ self._seeds = new_seeds
+
+ def save_as(self, filepath: Optional[PathType]):
+ """
+ Saves the seed to file. If the filepath is given, it will be used, and this will become the filepath
+ this SeedTable is associated to. Otherwise, the file to save to must have been specified as a setting
+ given at initialization of SeedTable(). If not, an exception will be raised. The seed file format is
+ same as in load().
+
+ :param filepath: the file to save to (will get overwritten if exists); this replaces the csv_path
+ set when the seed table was instantiated; if None, the file path will be based on the
+ csv_path that was set when the seed table was instantiated
+ """
+ if filepath is not None:
+ self._csv_path = self.get_csv_file_name(filepath)
+
+ log.info('Saving seeds to {}', self._csv_path)
+ with self._csv_path.open('w', newline='') as csv_file:
+ csv_writer = csv.writer(csv_file, dialect='excel')
+ csv_writer.writerow(self._csv_header)
+ for v_id, variant_seeds in enumerate(self._seeds):
+ if v_id >= MIN_VARIANT_ID:
+ for r_id, replic_seed in enumerate(variant_seeds):
+ if r_id >= MIN_REPLIC_ID:
+ csv_writer.writerow([v_id, r_id, replic_seed])
+
+ # --------------------------- instance PUBLIC properties and safe_slots ---------------------
+
+ num_variants = property(get_num_variants)
+ num_replics_per_variant = property(get_num_replics_per_variant)
+
+ # --------------------------- instance __SPECIAL__ method overrides -------------------------
+
+ def __len__(self):
+ """Number of seeds in this table"""
+ num_seeds = self._num_variants * self._replics_per_variant
+ assert len(self.get_seeds_list()) == num_seeds
+ return num_seeds
+
+ # --------------------------- instance _PROTECTED and _INTERNAL methods ---------------------
+
+ def _generate_seeds(self):
+ """Returns a 2D array of seeds (size (self._num_variants + 1) x (self._replics_per_variant + 1))"""
+ log.debug('Generating {}x{} unique seeds', self._num_variants, self._replics_per_variant)
+ # get one random seed per replication
+ rng = Random()
+ seeds = rng.sample(range(MIN_RAND_SEED, MAX_RAND_SEED), self._num_variants * self._replics_per_variant)
+ # put in 2D array and return it:
+ seeds_array = [[0] * (self._replics_per_variant + MIN_REPLIC_ID)]
+ for variant_id in range(1, self._num_variants + 1):
+ start_index = (variant_id - 1) * self._replics_per_variant
+ stop_index = variant_id * self._replics_per_variant
+ variant_seeds = seeds[start_index: stop_index]
+ variant_seeds.insert(0, 0)
+ seeds_array.append(variant_seeds)
+ assert seeds_array[variant_id] is variant_seeds
+ assert len(seeds_array[variant_id]) == self._replics_per_variant + 1
+ assert len(seeds_array) == self._num_variants + 1
+
+ return seeds_array
+
+ def _check_data_valid(self, variant: int, replic: int, seed: int):
+ """
+ Determine whether given arguments are valid (in range).
+ :param variant: The variant number to check
+ :param replic: The replication number to check
+ :param seed: The seed to check.
+ """
+ # TODO build 3: add test for this
+
+ if variant < MIN_VARIANT_ID:
+ raise ValueError("variant #{} must be >= 1".format(variant))
+
+ if replic < MIN_REPLIC_ID:
+ raise ValueError("replication #{} must be >= 1".format(replic))
+
+ if seed < MIN_RAND_SEED or seed > MAX_RAND_SEED:
+ msg = "random seed {} must be in range [{}, {}]".format(seed, MIN_RAND_SEED, MAX_RAND_SEED)
+ raise ValueError(msg)
diff --git a/origame/core/__init__.py b/origame/core/__init__.py
new file mode 100644
index 0000000..48e1faa
--- /dev/null
+++ b/origame/core/__init__.py
@@ -0,0 +1,40 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: This package provides core functionality used by various Origame components.
+
+In particular it defines BridgeEmitter and BridgeSignal: these default to being aliases for BackendEmitter
+and BackendSignal, respectively. Any component that uses those will automatically get a signal base class that
+is determined by the host application.
+
+Version History: See SVN log.
+"""
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5788$"
+
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- PUBLIC API ---------------------------------------------------------------------------------
+# import *public* symbols (classes/functions/constants) from contained modules:
+
+from .constants import *
+from .cmd_line_args_parser import RunScenCmdLineArgs, LoggingCmdLineArgs, ConsoleCmdLineArgs, BaseCmdLineArgsParser
+from .cmd_line_args_parser import AppSettings
+from .singleton import Singleton
+from .decorators import *
+from .base_fsm import BaseFsmState, IFsmOwner
+from .signaling import BackendEmitter, BackendSignal, BridgeEmitter, BridgeSignal, safe_slot
+from .utils import validate_python_name, get_valid_python_name, InvalidPythonNameError
+from .utils import UniqueIdGenerator, get_enum_val_name, select_object, ClockTimer, plural_if
+from ._logging import LogManager, LogRecord, LogCsvFormatter, log_level_int, log_level_name
+from .meta import AttributeAggregator
diff --git a/origame/core/_logging.py b/origame/core/_logging.py
new file mode 100644
index 0000000..b0cbbdd
--- /dev/null
+++ b/origame/core/_logging.py
@@ -0,0 +1,267 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Common logging functionality for Origami application variants.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import logging
+from pathlib import Path
+
+# [2. third-party]
+
+# [3. local]
+from .typing import Any, Either, Optional, Callable, PathType, TextIO, BinaryIO
+from .typing import List, Tuple, Sequence, Set, Dict, Iterable, Stream
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+__all__ = [
+ # defines module members that are public; one line per string
+ 'LogManager',
+ 'LogRecord',
+ 'LogCsvFormatter',
+ 'log_level_name',
+ 'log_level_int',
+]
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+log = logging.getLogger('system')
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+def log_level_name(level: Either[int, str]) -> str:
+ """
+ Returns the log level name for a given level. There is no equivalent in logging module:
+ logging.getLevelName(obj) returns the level value if obj is name, but the level name if obj is value.
+
+ :param level: int like logging.DEBUG, or string like 'DEBUG'
+ :returns: the corresponding string like 'DEBUG'
+
+ WARNING: uses instance, so do not call in compute intensive sections of code (e.g., loops).
+ """
+ return level if isinstance(level, str) else logging.getLevelName(level)
+
+
+def log_level_int(level: Either[int, str]) -> int:
+ """
+ Returns the log level integer value for a given level. There is no equivalent in logging module:
+ logging.getLevelName(obj) returns the level value if obj is name, but the level name if obj is integer value.
+
+ :param level: int like logging.DEBUG, or string like 'DEBUG'
+ :returns: the corresponding integer value like 10 for logging.DEBUG
+
+ WARNING: uses instance, so do not call in compute intensive sections of code (e.g., loops).
+ """
+ return level if isinstance(level, int) else logging.getLevelName(level)
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+class LogRecord(logging.LogRecord):
+ """Override logging.LogRecord's getMessage so that both .format and % are supported in log messages"""
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.csv_format = False
+
+ def getMessage(self):
+ """
+ Return the message for this LogRecord.
+
+ Return the message for this LogRecord after merging any user-supplied
+ arguments with the message.
+ """
+ msg = str(self.msg) # required per LogRecord docs
+ if self.args:
+ try:
+ msg = super().getMessage()
+ except:
+ msg = msg.format(*self.args)
+
+ if self.csv_format:
+ msg = msg.replace('"', '""')
+ return msg
+
+
+class LogManager:
+ """
+ Manages all log setup for a process. This should be instantiated once in the main process of both
+ Origame variants, and once by each background replication process.
+ """
+
+ def __init__(self, stream=None, log_level: Either[int, str] = None):
+ self._logfile = None
+
+ self._syslog = logging.getLogger('system')
+ self._usrlog = logging.getLogger('user')
+
+ # create new level
+ if not hasattr(logging, 'PRINT'):
+ self.setup_print_logger()
+
+ # set log levels for each logger:
+ if log_level is None:
+ log_level = logging.INFO
+ self._syslog.setLevel(log_level)
+ self._usrlog.setLevel(logging.PRINT)
+
+ # setup the logging to a stream if requested:
+ self._stream = None
+ if stream is not None:
+ # create a stream handler to stream to stream
+ self._stream = logging.StreamHandler(stream)
+ self._stream.setLevel(logging.DEBUG)
+ # Logs to the GUI log panel
+ formatter = logging.Formatter('%(asctime)s.%(msecs)03d:\t%(name)s:\t%(levelname)s:\t%(message)s',
+ datefmt='%m/%d/%Y %H:%M:%S')
+ self._stream.setFormatter(formatter)
+ self._syslog.addHandler(self._stream)
+ self._usrlog.addHandler(self._stream)
+
+ assert self.is_ready # post-condition
+
+ @staticmethod
+ def setup_print_logger():
+ logging.PRINT = 15
+ logging.addLevelName(logging.PRINT, 'PRINT')
+ userlog = logging.getLogger('user')
+
+ def userlog_print(msg, *args, **kwargs):
+ return userlog.log(logging.PRINT, msg, *args, **kwargs)
+
+ userlog.print = userlog_print
+
+ def log_to_file(self, path='.', filename='log.csv', create_path=False, write_mode='w'):
+ """
+ Call this when logging to file is desired. Can only be called once. Output format is CSV.
+ :param path: override default location of log file
+ :param filename: override default file name of log file
+ :param write_mode: specify the mode to open the file (for writing, reading, appending, etc.)
+ :raises: RuntimeError if called previously
+ :raises: OSError for problems encountered opening log file
+ """
+
+ assert self.is_ready # verify that we have not been closed already
+
+ if self._logfile is not None:
+ raise RuntimeError('Can only call this once per LogManager instance')
+
+ path = Path(path)
+ if create_path and not path.exists():
+ Path(path).mkdir(parents=True)
+ filepath = path / filename
+
+ # Clear previous contents and set up header in log file
+ header_line = 'Time [MM/DD/YYYY HH:MM:SS.mmm],Log Name,Log-Level,Message\n'
+ with filepath.open(write_mode) as f:
+ f.write(header_line)
+ f.close()
+
+ # Open the logfile using 'filepath' for 'appending' logs
+ self._logfile = logging.FileHandler(str(filepath)) # <- Note: this defaults to append mode
+ self._logfile.setLevel(logging.DEBUG)
+
+ formatter = LogCsvFormatter('{asctime},{name},{levelname},"{message}"', style='{')
+
+ formatter.default_time_format = '%m/%d/%Y %H:%M:%S'
+ formatter.default_msec_format = '%s.%03d'
+
+ self._logfile.setFormatter(formatter)
+
+ self._syslog.addHandler(self._logfile)
+ self._usrlog.addHandler(self._logfile)
+
+ def cleanup_files(self, glob_pattern: str, path: str = '.', keep: int = 5):
+ """
+ Cleanup existing log files that match glob_pattern. Folder containing log files is current working
+ director if not specified.
+ :param glob_pattern: pattern that will be given to glob.glob() function
+ :param path: folder in which to look for log files
+ :param keep: number of files to keep.
+ """
+ from pathlib import Path
+ file_list = Path(path).glob(glob_pattern)
+ file_list = sorted(file_list, key=lambda f: f.stat().st_mtime)
+ for file in file_list[:-keep]:
+ try:
+ file.unlink()
+ except IOError:
+ # file might be locked by other Origame, leave it alone
+ pass
+
+ def get_is_ready(self) -> bool:
+ """
+ Return true if this LogManager can be used: all loggers exist. This is the only method that can
+ be called after close().
+ """
+ return (self._syslog is not None) and (self._usrlog is not None)
+
+ def get_logfile_path(self) -> Path:
+ """Get the log file path, if any (None otherwise)"""
+ return self._logfile.baseFilename if self._logfile else None
+
+ def get_log_stream(self):
+ """Get the stream that was given at initialization, if any (None otherwise)"""
+ return self._stream.stream if self._stream else None
+
+ def close(self):
+ """
+ Shutdown this logging manager. In a multithreaded process (such as for testing batch_sim.Replication), need
+ to disconnect self from logging system without waiting for gc. Note: once called, only is_ready can be used.
+ """
+
+ if self._logfile is not None:
+ self._syslog.removeHandler(self._logfile)
+ self._usrlog.removeHandler(self._logfile)
+ self._logfile.close()
+ self._logfile = None
+
+ if self._stream is not None:
+ self._syslog.removeHandler(self._stream)
+ self._usrlog.removeHandler(self._stream)
+ self._stream = None
+
+ self._syslog = None
+ self._usrlog = None
+
+ def __del__(self):
+ self.close()
+
+ is_ready = property(get_is_ready)
+ logfile_path = property(get_logfile_path)
+ log_stream = property(get_log_stream)
+
+
+logging.setLogRecordFactory(LogRecord)
+
+
+class LogCsvFormatter(logging.Formatter):
+ """Formats log messages in CSV format for files, properly escaping messages that have commas"""
+
+ def format(self, record):
+ old_record_format = record.csv_format
+ record.csv_format = True
+ try:
+ return super().format(record)
+ finally:
+ record.csv_format = old_record_format
diff --git a/origame/core/base_fsm.py b/origame/core/base_fsm.py
new file mode 100644
index 0000000..8b054cc
--- /dev/null
+++ b/origame/core/base_fsm.py
@@ -0,0 +1,231 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Provide a generic Finite State Machine (FSM) base class for objects that have states
+
+An object that has states should derive from IFsmOwner and is called "the FSM owner". Each of its states should
+derive from BaseFsmState. The default state should be set directly by the FSM owner by setting self._state to
+an instance of the proper state class. Transitions are the responsibility of each state: it should call
+self._set_state(StateClass) for a transition to occur. This will automatically update the FSM owner with the
+new state. State-dependent methods of the FSM owner should delegate to the currently active state. Each
+state is assumed to have a numeric ID and string representation, typically achieved by creating a class that
+derives from IntEnum listing the possible state, and each state class defines state_id from one of these
+enum values.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import logging
+from enum import IntEnum
+
+# [2. third-party]
+
+# [3. local]
+from ..core import attrib_override_required, override_optional, override_required
+from ..core.typing import AnnotationDeclarations
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+__all__ = [
+ # public API of module: one line per string
+ 'BaseFsmState',
+ 'IFsmOwner'
+]
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+log = logging.getLogger('system')
+
+
+class Decl(AnnotationDeclarations):
+ BaseFsmState = 'BaseFsmState'
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+class IFsmOwner:
+ """
+ Any object that uses states should derive from this class. The _state attribute can be used by the derived
+ class to delegate method calls to the current state.
+ """
+
+ def __init__(self):
+ self._state = None
+
+ def is_state(self, state_id: IntEnum) -> bool:
+ """Return True if our current state has ID state_id, False otherwise"""
+ return state_id == self._state.state_id
+
+ def get_state_name(self) -> str:
+ """Return current state name"""
+ return self._state.state_name
+
+ def get_state_id(self) -> IntEnum:
+ """Return current state ID"""
+ return self._state.state_id
+
+ state_name = property(get_state_name)
+ state_id = property(get_state_id)
+
+ def _set_state(self, state: Decl.BaseFsmState):
+ """
+ Set the state of the FSM owner. After this is done, the _on_state_changed() is automatically called; derived
+ class may override as necessary to get notified of state changes.
+ :param state: new state
+ """
+ prev_state = self._state
+ self._state = state
+ self._on_state_changed(prev_state)
+
+ @override_optional
+ def _on_state_changed(self, prev_state: Decl.BaseFsmState):
+ """
+ Called automatically by _set_state() to indicate a state change. The new state is self._state.
+ :param prev_state: the previous state
+ """
+ pass
+
+
+class FsmOpUnavailable:
+ """
+ Return value proxy that automatically logs a warning when created and when called.
+ """
+ WARNING_IS_ERROR = True
+
+ class Error(RuntimeError):
+ pass
+
+ def __init__(self, state_name: str, attr_name: str):
+ msg = "state '{}' does not have an attribute called '{}'".format(state_name, attr_name)
+ if self.WARNING_IS_ERROR:
+ raise self.Error(msg)
+ else:
+ log.debug("WARNING: {}", msg)
+ self.__state_name = state_name
+ self.__attr_name = attr_name
+
+ def __call__(self, *args, **kwargs):
+ args = [str(arg) for arg in args]
+ kwargs = ['{}={}'.format(k, v) for k, v in kwargs.items()]
+ log.debug("WARNING: ignoring call of method {}({}) on state '{}'",
+ self.__attr_name, ', '.join(args + kwargs), self.__state_name)
+
+
+class BaseFsmState:
+ """
+ Base class for all states of a FSM.
+
+ Note:
+ - derived class must override state_id
+ - derived class can override the base versions of enter_state and exit_state.
+ - FSM owner must provide _set_state() method
+ """
+
+ state_id = attrib_override_required(None)
+
+ __state_name = None # used by get_state_name()
+
+ def __init__(self, prev_state: Decl.BaseFsmState, fsm_owner: IFsmOwner = None):
+ """
+ Initialize data for a new state. Set the previous state to None for the initial state, as this
+ automatically calls self.enter_state().
+
+ :param prev_state: previous state; if None, the enter_state() method is automatically called.
+ :param fsm_owner: the object that owns the FSM for this state. It is assumed to have a _state
+ attribute which will hold this instance.
+ """
+
+ assert self.state_id is not None
+
+ self._fsm_owner = fsm_owner
+ self._prev_state_class = prev_state.__class__
+ if prev_state is None:
+ log.debug('FSM {} initialized in {}', self.__get_fsm_owner_type_name(), self.state_id.name)
+ self.enter_state(None)
+
+ @override_optional
+ def enter_state(self, prev_state: Decl.BaseFsmState):
+ """
+ This will be called automatically by _set_state() after the FSM owner has had its state data member
+ reset to the new state, but before the previous state's exit_state() method is called. Most classes
+ don't need to override this, but can be useful if some actions should be taken only after the FSM
+ has new state.
+ :param prev_state: state that is being exited
+ """
+ pass
+
+ @override_optional
+ def exit_state(self, new_state: Decl.BaseFsmState):
+ """
+ States that need to take action on exit can override this to cleanup etc. It is called automatically
+ when a derived class calls _set_state, before entering new state.
+ :param new_state: state being transitioned to
+ """
+ pass
+
+ def get_state_name(self) -> str:
+ """Get the name for this state. It is the string rep of enumeration constant returned by state_id()."""
+ if self.__state_name is None:
+ self.__state_name = str(self.state_id).split('.')[-1]
+ return self.__state_name
+
+ state_name = property(get_state_name)
+
+ def __getattr__(self, attr_name: str):
+ """Attempt to get an attribute (to read it or call it as a function) that does not exist in this state"""
+ return FsmOpUnavailable(self.state_name, attr_name)
+
+ def _unsupported_op(self, op_name: str):
+ """
+ A derived class that is base to other states can call this method when a method it provides was
+ not overridden, indicating that the concrete state does not support it.
+ """
+ log.debug("WARNING: state '{}' does not have an attribute called '{}'", self.state_name, op_name)
+
+ def _set_state(self, state_class, **kwargs):
+ """
+ Transition to a new state. Derived class must call this to cause a transition to a new state.
+ The new state object is created from state_class, and set as fsm_data._state. Then _self.exit_state(new_state)
+ is called, followed by new_state.enter_state(). Finally, the sig_state_changed signal is emitted.
+
+ Note: If the current state cannot be exited, or the new state cannot be created or entered, the caller
+ of this method will have to decide what to do.
+
+ :param state_class: class to use for new state
+ :param **kwargs: arguments to give to state constructor
+ """
+ # first create the new state, and set in FSM owner
+ new_state = state_class(self, fsm_owner=self._fsm_owner, **kwargs) if state_class else None
+
+ # if this worked, can exit current state
+ log.debug("FSM {} exiting state {}", self.__get_fsm_owner_type_name(), self.state_name)
+ self.exit_state(new_state)
+
+ # and enter new state
+ log.debug("FSM {} entering state {}", self.__get_fsm_owner_type_name(), new_state.state_name)
+ self._fsm_owner._set_state(new_state)
+
+ new_state.enter_state(self)
+
+ def __get_fsm_owner_type_name(self) -> str:
+ """Get class name of FSM owner"""
+ return self._fsm_owner.__class__.__name__
diff --git a/origame/core/cmd_line_args_parser.py b/origame/core/cmd_line_args_parser.py
new file mode 100644
index 0000000..1b3df46
--- /dev/null
+++ b/origame/core/cmd_line_args_parser.py
@@ -0,0 +1,199 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: The command line argument parsing capability for the Origame application.
+
+This module contains the following classes:
+cmd_line_args_parser: A wrapper class for the argparse module that customizes commandline parsing for the Origame
+ application.
+SimConfig: A helper class defined within the cmd_line_args_parser class. Used to store commandline arguments parsed by
+ the parent class.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+from argparse import ArgumentParser, Namespace
+
+# [2. third-party]
+
+# [3. local]
+from .typing import Any, Either, Optional, Callable, PathType, TextIO, BinaryIO
+from .typing import List, Tuple, Sequence, Set, Dict, Iterable, Stream
+from .decorators import override
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+__all__ = [
+ # public API of module
+ 'RunScenCmdLineArgs',
+ 'LoggingCmdLineArgs',
+ 'ConsoleCmdLineArgs',
+ 'BaseCmdLineArgsParser',
+]
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+class LoggingCmdLineArgs(ArgumentParser):
+ """
+ Common command-line args functionality for all Origame variants.
+ """
+
+ def __init__(self):
+ super().__init__(add_help=False)
+
+ # NOTE: each argument must have a default!
+ self.add_argument("--dev-log-level",
+ dest='log_level', type=str, default=None, # default so we know if user wants custom
+ help="Log level for application (does not propagate to batch replications): "
+ "DEBUG, INFO, WARNING, ERROR, CRITICAL.")
+ self.add_argument("--dev-no-save-log",
+ dest='save_log', default=True, action='store_false',
+ help="Do not save log to a file")
+ self.add_argument("--dev-no-dep-warn",
+ dest='log_deprecated', default=True, action='store_false',
+ help="Turn off logging of prototype API deprecation warnings")
+ self.add_argument("--dev-log-raw-events",
+ dest='log_raw_events', default=False, action='store_true',
+ help="Turn on logging of sim event push/pops raw data (at warn level)")
+ self.add_argument("--dev-no-linking-fixes-on-load",
+ dest='fix_linking_on_load', default=True, action='store_false',
+ help="Turn off fixing of invalid links on scenario load")
+
+
+class RunScenCmdLineArgs(ArgumentParser):
+ """
+ Command-line arguments related to running a scenario.
+ """
+
+ def __init__(self):
+ super().__init__(add_help=False)
+
+ self.add_argument("scenario_path", type=str,
+ help="Scenario definition file pathname. Mandatory. Can be relative path.")
+
+ # NOTE: each remaining argument must have a default!
+
+ # common to batch and non-batch:
+ self.add_argument("--loop-log-level",
+ type=str, default=None, # this is how we know whether user overrides the app default
+ help="Log level for each replication's sim loop: DEBUG, INFO, WARNING, ERROR, CRITICAL.")
+ self.add_argument("-t", "--max-sim-time-days",
+ type=float, default=None,
+ help="Simulation cut-off time in days. Optional. Default runs till no events (or max "
+ "real-time reached, if set; whichever occurs first).")
+ self.add_argument("-x", "--max-wall-clock-sec",
+ type=float, default=None,
+ help="Real-time cut-off time in seconds. Optional. Default runs till no events (or "
+ "max sim time reached, if set; whichever occurs first).")
+
+ # only in non-batch mode:
+ self.add_argument("-f", "--realtime-scale",
+ type=float, default=None,
+ help="Turn on real-time mode, using the given scale factor. Default is "
+ "as-fast-as-possible.")
+
+ # only in batch mode:
+ self.add_argument("-b", "--batch-replic-save",
+ default=True, action='store_false',
+ help="Turn off saving of final scenario state by each batch replication")
+ self.add_argument("-s", "--seed-file-path",
+ type=str, default=None,
+ help="Seed file pathname. Optional. Can be relative path. Default causes seeds to be "
+ "randomly generated.")
+ self.add_argument("-v", "--num-variants",
+ type=int, default=1,
+ help="The number of scenario variants to be run. Optional. Default runs a single "
+ "variant.")
+ self.add_argument("-r", "--num-replics-per-variant",
+ type=int, default=1,
+ help="The number of scenario replications to be run for a scenario variant. "
+ "Optional. Default runs a single replication for a variant.")
+ self.add_argument("-c", "--num-cores",
+ type=int, default=0,
+ help="The maximum number of cores to utilize for the configured scenario run. "
+ "Optional. Zero distributes batch replications across all available cores.")
+
+
+class BaseCmdLineArgsParser(ArgumentParser):
+ @classmethod
+ def get_defaults(cls, *required: List[str], dest: Namespace = None) -> Namespace:
+ """
+ Get the default settings.
+ :param required: values for command line args that are required (don't have defaults)
+ :param dest: If dest given, attributes are created in dest, else a new namespace is returned
+ """
+ return cls().parse_args(args=required, namespace=dest)
+
+ def parse_args(self, args=None, namespace=None):
+ """Once the namespace is populated, protect it against changing settings directly"""
+ if namespace is None:
+ namespace = AppSettings()
+ result = ArgumentParser.parse_args(self, args=args, namespace=namespace)
+ result.protected = True
+ return result
+
+
+class ConsoleCmdLineArgs(BaseCmdLineArgsParser):
+ def __init__(self):
+ log_clap = LoggingCmdLineArgs()
+ run_scen_clap = RunScenCmdLineArgs()
+ super().__init__(parents=[log_clap, run_scen_clap])
+
+ self.add_argument("-l", "--dev-no-stdout-log",
+ dest='console_logging', default=True, action='store_false',
+ help="Log system debug messages (assuming logging is ON)")
+
+
+class AppSettings:
+ """
+ Application settings object created from command line arguments. Which settings it contains
+ depends on the command line args parser; the parser will populate it with all the command
+ line argument values as well as the default values (for command line args not used).
+
+ Since an instance is basically a data structure, it is very easy to change settings (say for
+ testing) but this also means that typos will go unnoticed and the default value will be used
+ (which will typically be difficult bug to figure out as it will appear as though the setting
+ has no effect). For this reason, once the command line args parser is done creating the
+ settings object, it sets it in protected mode: the only way to further change settings after
+ this is via the override() method, which accepts a setting only if it exists already in the
+ instance.
+ """
+
+ def override(self, **kwargs):
+ """
+ Override existing settings. The settings MUST have been obtained from one of the command
+ line args parsers
+ """
+ if not set(kwargs.keys()).issubset(self.__dict__):
+ unknown_keys = [repr(s) for s in set(kwargs.keys()).difference(self.__dict__)]
+ raise ValueError('Invalid kwarg names: {} (valid are {})'.format(
+ ', '.join(unknown_keys), ', '.join(sorted(self.__dict__))))
+
+ self.__dict__.update(kwargs)
+
+ def __setattr__(self, key, value):
+ """Prevent directly settings attributes when protected so that override() MUST be used"""
+ if key != 'protected' and hasattr(self, 'protected'):
+ raise RuntimeError('Settings protected, use override(**settings) method')
+
+ object.__setattr__(self, key, value)
diff --git a/origame/core/constants.py b/origame/core/constants.py
new file mode 100644
index 0000000..f8bf56e
--- /dev/null
+++ b/origame/core/constants.py
@@ -0,0 +1,55 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: This module defines constants and conversion factors common to the entire Origame application.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import logging
+
+# [2. third-party]
+
+# [3. local]
+
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+
+__all__ = [
+ # public API of module: one line per string
+ 'SECONDS_PER_DAY',
+ 'MINUTES_PER_DAYS',
+ 'HOURS_PER_DAYS',
+ 'SECONDS_TO_DAYS',
+ 'MINUTES_TO_DAYS',
+ 'HOURS_TO_DAYS',
+]
+
+# Time conversion factors
+SECONDS_PER_DAY = 86400.0
+MINUTES_PER_DAYS = 1440.0
+HOURS_PER_DAYS = 24.0
+SECONDS_TO_DAYS = 1.0 / SECONDS_PER_DAY
+MINUTES_TO_DAYS = 1.0 / MINUTES_PER_DAYS
+HOURS_TO_DAYS = 1.0 / HOURS_PER_DAYS
+
+# -- Function definitions -----------------------------------------------------------------------
+
+# -- Class Definitions --------------------------------------------------------------------------
diff --git a/origame/core/decorators.py b/origame/core/decorators.py
new file mode 100644
index 0000000..86e2d77
--- /dev/null
+++ b/origame/core/decorators.py
@@ -0,0 +1,151 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Decorators for various uses
+
+Decorators for various uses, including annotating function/method signatures. For use anywhere in Origame.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import inspect
+
+# [2. third-party]
+
+# [3. local]
+from .typing import TypeVar, Any, Either, Optional, Callable, PathType, TextIO, BinaryIO
+from .typing import List, Tuple, Sequence, Set, Dict, Iterable, Stream
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+__all__ = [
+ # defines module members that are public; one line per string
+ 'override',
+ 'override_required',
+ 'override_optional',
+ 'attrib_override_required',
+ 'internal',
+]
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+TAny = TypeVar('TAny')
+TCallable = Callable[[Any], Any] # any callable is accepted
+Decorator = Callable[[TCallable], TCallable]
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+def override(base_class: type) -> Decorator:
+ """
+ Indicate that a method is an override of a base class method. For documentation, and checks that the method
+ indeed exists in the base class. If not, an assertion error will arise at import time. Example:
+
+ >>> class Foo:
+ ... def base_meth(self):
+ ... print('foo')
+ ...
+ >>> class Bar(Foo):
+ ... @override(Foo) # if Foo.base_meth does not exist, AssertError will be raised at import time
+ ... def base_meth(self):
+ ... Foo.base_meth(self)
+ ... print('bar')
+ ...
+ >>> bar = Bar()
+ >>> bar.base_meth()
+ foo
+ bar
+ >>> print('name:', bar.base_meth.__name__)
+ name: base_meth
+ """
+ if not inspect.isclass(base_class):
+ raise ValueError('Need base class to be given as arg to decorator')
+
+ def check_derived_method(fn):
+ func_name = fn.__name__
+ if not hasattr(base_class, func_name):
+ err_msg = "Derived method '{}' is not in {}".format(func_name, base_class.__qualname__)
+ raise AttributeError(err_msg)
+ return fn
+
+ return check_derived_method
+
+
+def override_required(func: TCallable) -> TCallable:
+ """
+ Use this decorator to indicate that a method MUST be overridden in a derived class. The based class method
+ should raise NotImplementedError to ensure that calling a non-overridden override_required doesn't go unnoticed.
+ """
+ return func
+
+
+def override_optional(func: TCallable) -> TCallable:
+ """
+ Use this decorator to indicate that a method can safely be overridden in a derived class. The base class
+ method should provide valid default behavior.
+ """
+ return func
+
+
+def attrib_override_required(default_val: TAny) -> TAny:
+ """
+ When a base class attribute is marked with this "decorator", it MUST be overridden by the derived class.
+ There is currently no way to enforce this, so this is just a means to make the base class API contract explicit.
+ :return: default_val
+ """
+ return default_val
+
+
+def internal(*types: List[type]) -> Decorator:
+ """
+ Decorator to indicate that a function should be treated as public only to the types given or, if no
+ types given, to types defined within the same module; the function should be treated as private for
+ everything else. Note: The decorator has no way of enforcing that access to the decorated function is
+ only through specified classes (if specified) or classes of the same module; it can only make the
+ intent clear to the caller.
+
+ The decorated function must start with one (and only one) underscore (or a ValueError is raised).
+
+ Example:
+
+ >>> class Foo:
+ ... @internal(Baz) # _meth should be accessed only by Baz, assumed to be in same module or package
+ ... def _meth(self):
+ ... pass
+ ...
+ >>>
+ """
+
+ def check_valid_name(func: Callable):
+ func_name = func.__name__
+ if not func_name.startswith('_') or func_name.startswith('__'):
+ raise ValueError('All internal function names must start with one and only one underscore')
+
+ # handle the case where no types were specified:
+ if len(types) == 1 and type(types[0]).__name__ in ('function',):
+ method = types[0]
+ check_valid_name(method)
+ return method
+
+ # when types specified, the return must be the "actual" decorator that will wrap the function:
+ def decorator(func):
+ check_valid_name(func)
+ return func
+
+ return decorator
diff --git a/origame/core/meta.py b/origame/core/meta.py
new file mode 100644
index 0000000..f74c0ce
--- /dev/null
+++ b/origame/core/meta.py
@@ -0,0 +1,124 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Meta classes and meta programming support.
+
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import logging
+
+# [2. third-party]
+
+# [3. local]
+from .typing import Any, Either, Optional, Callable, PathType, TextIO, BinaryIO
+from .typing import List, Tuple, Sequence, Set, Dict, Iterable, Stream
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+
+__all__ = [
+ # public API of module: one line per string
+ 'AttributeAggregator'
+]
+
+log = logging.getLogger('system')
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+
+class AttributeAggregator(type):
+ """
+ This class is a metaclass. It is used by those classes that need to combine attributes consistently.
+
+ This takes the META_AUTO_*_API_EXTEND class attributes of a class being
+ imported and combines them into the class's AUTO_*_API_CUMUL attribute. Hence each class in the part type hierarchy
+ ends up with its own AUTO_*_API_CUMUL attribute that contains the EXTEND items of all levels "up". The AUTO
+ attributes are used by BaePart to provide functionality that is common to all parts, such as editing
+ a part, searching its properties, accessing its scripting API, and so forth.
+
+ Example: if class A derives from B which derives from BasePart, both A and B can define
+ META_AUTO_SCRIPTING_API_EXTEND. B.META_AUTO_SCRIPTING_API_EXTEND contains, as per the META_ docs in BasePart,
+ references to B members that should be available via auto-completion when scripting, such as B.bar1 and B.bar2;
+ A.META_AUTO_SCRIPTING_API_EXTEND does the same, such as A.aaa1 and A.aaa2. This this meta class will cause
+ B.AUTO_SCRIPTING_API_CUMUL to be ('bar1', 'bar2'), and A.AUTO_SCRIPTING_API_CUMUL to be
+ ('bar1', 'bar2', 'aaa1', 'aaa2'). The META_*_EXTEND members are deleted from the class once processed,
+ so that optional METAs can be ommitted:
+
+ - META_AUTO_EDITING_API_EXTEND and META_AUTO_SCRIPTING_API_EXTEND: required
+ - META_AUTO_SEARCHING_API_EXTEND: if not given, AUTO_SEARCHING_API_CUMUL will be AUTO_EDITING_API_CUMUL
+ - META_AUTO_ORI_DIFFING_API_EXTEND: if not given, AUTO_ORI_DIFFING_CUMUL will be AUTO_SEARCHING_API_CUMUL
+ """
+
+ def __init__(cls, name, bases, *args):
+ """
+ :param cls: the class that is being imported, and is ready for final initialization (ex FunctionPart, etc)
+ :param name: the class name
+ :param bases: the base classes of the class
+ :param args: unused arguments
+ """
+ assert hasattr(cls, 'META_AUTO_EDITING_API_EXTEND')
+
+ cls.__combine_api_traits(bases, 'META_AUTO_EDITING_API_EXTEND',
+ 'AUTO_EDITING_API_CUMUL')
+ cls.__combine_api_traits(bases, 'META_AUTO_SEARCHING_API_EXTEND',
+ 'AUTO_SEARCHING_API_CUMUL', 'AUTO_EDITING_API_CUMUL')
+ cls.__combine_api_traits(bases, 'META_AUTO_ORI_DIFFING_API_EXTEND',
+ 'AUTO_ORI_DIFFING_CUMUL', 'AUTO_SEARCHING_API_CUMUL')
+ cls.__combine_api_traits(bases, 'META_AUTO_SCRIPTING_API_EXTEND',
+ 'AUTO_SCRIPTING_API_CUMUL')
+
+ def __combine_api_traits(cls, bases: List[type], trait_members_list_name: str, cumul_name: str,
+ default_cumul_name: str = None):
+ base_cumul_list = []
+ for base in bases:
+ if hasattr(base, cumul_name):
+ base_cumul_list += getattr(base, cumul_name)
+
+ try:
+ trait_members = getattr(cls, trait_members_list_name)
+ # must delete so that optional METAs supported:
+ delattr(cls, trait_members_list_name)
+
+ except AttributeError:
+ if default_cumul_name is None:
+ raise ValueError('Class {} does not define required "{}"'.format(cls.__name__, trait_members_list_name))
+ setattr(cls, cumul_name, getattr(cls, default_cumul_name))
+
+ else:
+ cls_cumul_list = []
+ for item in trait_members:
+ attrib_names = dir(cls)
+ found = False
+ for attrib_name in attrib_names:
+ if getattr(cls, attrib_name) is item:
+ cls_cumul_list.append(attrib_name)
+ found = True
+ break
+
+ if not found:
+ raise RuntimeError('BUG: attrib "{}" could not be matched to an object in class {}'
+ .format(attrib_name, cls.__name__))
+
+ assert len(cls_cumul_list) == len(trait_members)
+ setattr(cls, cumul_name, tuple(base_cumul_list + cls_cumul_list))
diff --git a/origame/core/signaling.py b/origame/core/signaling.py
new file mode 100644
index 0000000..012dc8e
--- /dev/null
+++ b/origame/core/signaling.py
@@ -0,0 +1,247 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Provide raw signaling class that mimics PyQt's QObject class
+
+Backend objects that do not interface with UI can derive from BackendEmitter and define BackendSignals.
+Those that do should derive from BridgeEmitter and define BridgeSignal. Those objects will automatically
+get the correct signaling baseclass as determined by the UI.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import logging
+
+# [2. third-party]
+
+# [3. local]
+from .decorators import override_optional
+from .typing import Any, Either, Optional, TypeVar, Callable, PathType, TextIO, BinaryIO
+from .typing import List, Tuple, Sequence, Set, Dict, Iterable, Stream
+from .typing import AnnotationDeclarations
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+__all__ = [
+ 'BackendEmitter',
+ 'BackendSignal',
+ 'BridgeEmitter',
+ 'BridgeSignal',
+ 'safe_slot',
+]
+
+log = logging.getLogger('system')
+
+TCallable = Callable[[Any], Any] # any callable is accepted
+
+
+class Decl(AnnotationDeclarations):
+ BackendEmitter = 'BackendEmitter'
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+def safe_slot(fn: TCallable) -> TCallable:
+ """
+ Return a "safetied" wrapper of fn, suitable for connecting to signals. This is needed only to match
+ the API of signaling systems used in GUI, since core and scenario use Qt signals and slots when imported
+ in GUI variant.
+ """
+ return fn
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+class _BackendSignalBound:
+ """
+ This should never be instantiated directly: bound signals can only be created by object that derive
+ from BackendEmitter, which does this automatically. The methods are documented in the unbound version of
+ this class.
+ """
+
+ def __init__(self, types: List[type]):
+ self.__slots = []
+ self.__types = types
+
+ def connect(self, slot: Callable):
+ if slot not in self.__slots:
+ self.__slots.append(slot)
+
+ def emit(self, *data: List[Any]):
+ """Call each slot with the provided data"""
+ for slot in self.__slots:
+ slot(*data)
+
+ def disconnect(self, slot: Callable = None):
+ """Disconnect provided slot. If no slot provided, disconnect all slots from this signal."""
+ if slot is None:
+ self.__slots = []
+ elif slot in self.__slots:
+ self.__slots.remove(slot)
+
+ def __call__(self, *args):
+ """Support chaining of signals (using a signal as a slot for another signal)"""
+ self.emit(*args)
+
+
+class BackendSignal:
+ """
+ Represent a (unbound) Backend Signal object. This class is designed to be a drop-in replacement for
+ pyqtSignal when the class that emits the signal must work with PyQt QObject.
+
+ As with PyQt signals, define an *instance* of this class in a class that
+ emits pure backend signals (i.e., signals that never connect to GUI objects). The emitting instance
+ must derive from BackendEmitter.
+
+ Note: The BackendEmitter creates a bound signal for the instance, from the class-wide
+ unbound signal, whenever the signal is accessed on the instance. Example:
+
+ class Foo(BackendEmitter):
+ sig_something = BackendSignal() # unbound
+
+ def update():
+ self.sig_something.emit() # bound signal, created at runtime
+
+ A "Foo.sig_something" refers to class-wide unbound signal object. But as soon as code uses "foo.sig_something",
+ where foo is an instance of Foo, BackendEmitter intercepts this to create a BridgeSignalBound instance on foo.
+ This is same strategy as used by PyQt for pyQtSignal's), with which BackendSignal must be compatible. Unbound
+ signals do not have implementations for emit, connect or disconnect. However,
+ in order to provide for code completion from IDE, stub methods are provided for these.
+ """
+
+ def __init__(self, *types: List[type]):
+ """
+ :param types: array of types accepted as payload when emitted
+ """
+ self.__types = types
+ self.__bound_signals = {}
+
+ def __get__(self, obj: Any, obj_type: type) -> _BackendSignalBound:
+ """
+ Intercept access to unbound signal on an object and create a signal bound to the object.
+ Note: NO LONGER USED. A different technique is used that requires BackendEmitter initialization
+ but is faster. It is kept here because it is too early to tell if the other technique
+ has limitations.
+
+ :param obj: the object derived from BackendEmitter
+ :param obj_type: should be BackendEmitter
+ :return: the new bound signal
+ """
+ if obj is None:
+ # we are being called on a class, nothing to do:
+ return self
+
+ if obj in self.__bound_signals:
+ # we have already been called on this instance, return the cached signal
+ return self.__bound_signals[obj]
+
+ # return new bound signal, after caching
+ obj_signal = _BackendSignalBound(self.__types)
+ self.__bound_signals[obj] = obj_signal
+ return obj_signal
+
+ def new_bound(self) -> _BackendSignalBound:
+ return _BackendSignalBound(self.__types)
+
+ def connect(self, slot: Callable):
+ """Connect this signal to given slot, which an be any callable. The callable will be called
+ with the arguments given to emit(). """
+ raise NotImplementedError('BackendSignal.connect() not implemented by bound signal')
+
+ def emit(self, *args: List[Any]):
+ """Emit the signal, with given arguments. The arguments should have the same types as the sequence
+ given to initializer of this class. """
+ raise NotImplementedError('BackendSignal.emit() not implemented by bound signal')
+
+ def disconnect(self, slot: Callable = None):
+ """Disconnect this signal from given slot, or from all slots if none given. """
+ raise NotImplementedError('BackendSignal.disconnect() not implemented by bound signal')
+
+
+class BackendEmitter:
+ """
+ Base class for any signal-emitting object in the backend of Origame. Derive from this class and
+ define signals as class-wide BackendSignal instances. Notes:
+ - some methods on this class are provided strictly so BackendEmitter can be a drop-in replacement for
+ QObject when application does not have a GUI.
+ -
+ """
+
+ __signals = {} # will hold a reference to each unbound signal defined on derived classes
+
+ def __init__(self, emitter_parent: Decl.BackendEmitter = None, thread: Any = None, thread_is_main: bool = None):
+ """
+ Derived class must initialize base: this will find all class-wide BackendSignal instances and
+ create instance-specific BackendSignalBound objects that can be used to connect to, emit, and disconnect
+ from slots.
+ """
+ assert thread is None, "The caller is probably expecting to be using BridgeEmitter, not BackendEmitter"
+ assert thread_is_main in (None, False, True)
+ self._emitter_parent = emitter_parent
+
+ if BackendEmitter.__signals:
+ for (attr, value) in self.__signals.items():
+ setattr(self, attr, value.new_bound())
+
+ else:
+ for (attr, value) in vars(self.__class__).items():
+ if isinstance(value, BackendSignal):
+ BackendEmitter.__signals[attr] = value
+ setattr(self, attr, value.new_bound())
+
+ def getParent(self) -> Decl.BackendEmitter:
+ return self._emitter_parent
+
+ def moveToThread(self, thread):
+ raise NotImplementedError('BackendEmitter.moveToThread() override function has not been implemented.')
+
+ def thread(self):
+ raise NotImplementedError('BackendEmitter.thread() override function has not been implemented.')
+
+ def deleteLater(self):
+ raise NotImplementedError('BackendEmitter.deleteLater() override function has not been implemented.')
+
+ def startTimer(self, event: Any) -> int:
+ raise RuntimeError("This should never be called in console variant")
+
+ @override_optional
+ def timerEvent(self, event: Any):
+ """This method will only be called if startTimer() was called from GUI. Never called in Console."""
+ raise NotImplementedError('BackendEmitter.timerEvent() override function has not been implemented.')
+
+ def killTimer(self, id: int):
+ raise RuntimeError("This should never be called in console variant")
+
+
+"""Class to use for signals that *could* interface with UI objects when a module is imported in the GUI variant. """
+BridgeSignal = BackendSignal
+
+"""Base class to use for classes that use BridgeSignal."""
+BridgeEmitter = BackendEmitter
+
+
+def setup_bridge_for_console():
+ """Used only during testing: switch back to using backend classes for bridge."""
+ global BridgeSignal, BridgeEmitter
+ BridgeSignal = BackendSignal
+ BridgeEmitter = BackendEmitter
+
+ from .. import core
+ core.BridgeSignal = BackendSignal
+ core.BridgeEmitter = BackendEmitter
diff --git a/origame/core/singleton.py b/origame/core/singleton.py
new file mode 100644
index 0000000..a5cb255
--- /dev/null
+++ b/origame/core/singleton.py
@@ -0,0 +1,62 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Provides a Singleton class
+
+Classes that represent singletons should derive from it.
+Users create instances normally but get the same instance instead.
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+
+# [2. third-party]
+
+# [3. local]
+
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+__all__ = [
+ # defines module members that are public; one line per string
+ 'Singleton',
+]
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+
+class Singleton:
+ """
+ Derive from this class to make the derived class a singleton. Then any derived class
+ instantiation is actually getting the same object. Example:
+
+ >>> class Foo(Singleton): pass
+ >>> foo = Foo()
+ >>> foo2 = Foo()
+ >>> assert foo is foo2
+ >>> assert id(foo) == id(foo2)
+ """
+
+ _instance = None
+
+ def __new__(cls, *args, **kwargs):
+ if not cls._instance:
+ cls._instance = super(Singleton, cls).__new__(cls, *args, **kwargs)
+ return cls._instance
diff --git a/origame/core/typing.py b/origame/core/typing.py
new file mode 100644
index 0000000..cc0412d
--- /dev/null
+++ b/origame/core/typing.py
@@ -0,0 +1,104 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Annotation symbols standard in R4.
+
+See docs for the standard "typing" and "collections.abc" modules.
+Use type aliases when the annotation is too verbose::
+
+ Range = Either[int, str, slice, List[int]]
+
+ def func(abc: Range) -> int:
+ pass
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+from typing import Any # any type of object
+from typing import Optional # Ex: def func(abc: Optional[int]): a can be int or None
+from typing import Callable # Ex: def func(on_done: Callable[[int, str], List[int]]: on_done is a callable
+ # that takes an int and str (in that order), and returns a list of integers
+
+from typing import TextIO # Ex: def func(writer: TextIO): writer is an object that has write(str)
+from typing import BinaryIO # Ex: def func(writer: BinaryIO): writer is an object that has write(bytes)
+
+from typing import Tuple # Ex: def func(a1: Tuple[int, int, str]): a1 is a tuple with an 2 ints and a str
+from typing import List, Sequence # Ex: def func(a1: List[int], a2: Sequence[str]): a1 is a list of integers,
+ # whereas a2 can be a list or a tuple or other iteratable object
+from typing import Set, FrozenSet # Ex: def func(s1: Set[int], s2: FrozenSet[str]): s1 is a set() of integers,
+ # s2 is a frozenset() of strings
+from typing import NamedTuple # Ex: def func(nt: NamedTuple('Employee', [('name', str), ('id', int)])):
+ # nt is a collections.namedtuple('Employee', ['name', 'id'])
+from typing import Iterable # iterable is a generic, dynamically generated sequence of items
+ # Ex: def func(iter: Iterable): iter can be used in a for loop/list comprehension
+ # as many times as desired
+from typing import Generator # a function that has yield
+
+from typing import Dict # type is dict; Ex: def func(d1: Dict[int, str]): d1 maps ints to strings
+from typing import KeysView # Ex: def func(m1: KeysView[int, str]): m1 is a Dict[int, str].keys()
+from typing import ValuesView # Ex: def func(m1: ValuesView[int, str]): m1 is a Dict[int, str].values()
+from typing import ItemsView # Ex: def func(m1: ItemsView[int, str]): m1 is a Dict[int, str].items()
+
+from typing import Generic, TypeVar
+
+# [2. third-party]
+
+# [3. local]
+
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 6971 $"
+
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+# the following lines are solely so PyCharm can find "Either" as a symbol and provide code completion
+from typing import Union as _Union, Iterator as _Iterator
+
+Either = _Union # Ex: def func(abc: Either[int, str, None]): abc can be int or str or None
+Stream = _Iterator # iterable that can only be used once (usually because it was created by calling a
+ # generator function (a function that yields); once loop to end, can't loop again
+
+from pathlib import Path
+PathType = Either[str, Path]
+
+
+class AnnotationDeclarations:
+ """
+ Used to define *forward declarations* for type hints (annotations).
+ These are only needed where a class defines methods that take and/or returns objects of its own
+ type, or when two objects' type hints need to refer to each other. Example:
+
+ # in the globals section of a module:
+
+ class Decl(AnnotationDeclarations):
+ Foo = 'Foo'
+ Bar = 'Bar'
+
+ # further down in the module:
+
+ class Foo:
+ def method(self, foo: Decl.Foo, bar: Decl.Bar):
+ ...
+
+ class Bar:
+ def method(self, foo: Decl.Foo, bar: Decl.Bar):
+ ...
+
+ The AnnotationDeclarations base class must be used and will ensure (at import type) that the derived
+ class name is Decl and that all symbols match their string value.
+ """
+ pass
\ No newline at end of file
diff --git a/origame/core/utils.py b/origame/core/utils.py
new file mode 100644
index 0000000..17dc66d
--- /dev/null
+++ b/origame/core/utils.py
@@ -0,0 +1,601 @@
+# This file is part of Origame. See the __license__ variable below for licensing information.
+#
+# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
+# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+#
+# For coding standards that apply to this file, see the project's Coding Standards document,
+# r4_coding_standards.html, in the project's docs/CodingStandards/html folder.
+
+"""
+*Project - R4 HR TDP*: Various core utility functions and classes
+
+Version History: See SVN log.
+"""
+
+# -- Imports ------------------------------------------------------------------------------------
+
+# [1. standard library]
+import logging
+import re
+import keyword
+from datetime import timedelta
+from time import time
+from cProfile import Profile
+from enum import Enum
+from pathlib import Path
+import sched
+
+# [2. third-party]
+from dateutil.relativedelta import relativedelta
+
+# [3. local]
+from .typing import Any, Either, Optional, Callable, TypeVar, PathType, TextIO, BinaryIO
+from .typing import List, Tuple, Sequence, Set, Dict, Iterable, Stream
+
+# -- Meta-data ----------------------------------------------------------------------------------
+
+__version__ = "$Revision: 5800$"
+__license__ = """This file can ONLY be copied, used or modified according to the terms and conditions
+ described in the LICENSE.txt located in the root folder of the Origame package."""
+__copyright__ = "(c) Her Majesty the Queen in Right of Canada"
+
+# -- Module-level objects -----------------------------------------------------------------------
+
+
+__all__ = [
+ # public API of module: one line per string
+ 'validate_python_name',
+ 'get_valid_python_name',
+ 'InvalidPythonNameError',
+ 'get_enum_val_name',
+ 'UniqueIdGenerator',
+ 'BlockProfiler',
+ 'ori_profile',
+ 'ClockTimer',
+ 'plural_if',
+ 'bool_to_not',
+ 'GuardFlag',
+ 'rel_to_timedelta',
+ 'timedelta_to_rel',
+]
+
+log = logging.getLogger('system')
+
+TAny = TypeVar('TAny')
+TCallable = Callable[[Any], Any] # any callable is accepted
+
+
+# -- Function definitions -----------------------------------------------------------------------
+
+def get_verified_eval(value_in_str: str):
+ """
+ Checks if a string can pass eval(). If it can, the eval() value will be returned; otherwise a SyntaxError will be
+ thrown.
+ :param value_in_str: The string to be checked
+ :return: The return value of the eval(value_in_str)
+ :raises: whatever exception is raised by eval(); an additional field 'text' is created, holding value_in_str
+ """
+ try:
+ obj_verified = eval(value_in_str)
+ except Exception as e:
+ e.text = value_in_str
+ raise
+
+ return obj_verified
+
+
+def get_verified_repr(val: Any) -> str:
+ """
+ Getting a verified repr from val means eval(repr(val)) == val is true.
+ :param val: The object from which the repr is derived
+ :return: The repr on the val
+ :raise: Exception if repr() or eval() fails.
+ :raise: RuntimeError if eval(repr(val)) == val is false.
+ """
+ cell_repr = repr(val)
+ cell_eval = eval(cell_repr)
+
+ if val != cell_eval:
+ raise RuntimeError("The string representation of the given value cannot pass eval()")
+
+ return cell_repr
+
+
+def validate_python_name(name: str):
+ """
+ This function validates whether or not the input name is a valid Python name. If the name is valid, the function
+ returns; however, if the input name is invalid, an InvalidPythonNameError is raised describing the format error
+ contains a proposal for a corrected version of the name.
+
+ Invalid names are names that: (Note: Corresponding auto-corrections applied are shown in brackets.)
+ - Are reserved keywords (prepend 'Obj_' to name)
+ - Start with a number (prepend underscore '_' to name)
+ - Contain any non-alpha-numeric character except the underscore (replace invalid character with underscore '_')
+
+ :param name: The name to be validated.
+ :raises: InvalidPythonNameError - An exception containing a description of the error and a proposed corrected
+ name.
+ """
+ # No name case
+ if name is None:
+ raise InvalidPythonNameError(msg='Name cannot be None', invalid_name=name, proposed_name='unnamed')
+
+ # This case covers a valid name
+ regex_py_lex = '[_A-Za-z][_a-zA-Z0-9]*'
+ if re.fullmatch(regex_py_lex, name) and not keyword.iskeyword(name):
+ return
+
+ # For invalid names, the following apply
+
+ # Invalid case: keyword used (prepend 'Obj_')
+ if keyword.iskeyword(name):
+ new_name = 'Obj_{}'.format(name)
+ raise InvalidPythonNameError(
+ msg="Name cannot be a Python keyword.", invalid_name=name, proposed_name=new_name)
+
+ # Invalid case: invalid characters used
+ # - numeric character is the first character (prepend '_')
+ # - non-alpha-numeric character(s) used (replace with '_')
+
+ bad_format = False
+ orig_name = name
+ regex_num_first = '[0-9]'
+ if re.fullmatch(regex_num_first, name[0]):
+ name = '_{}'.format(name)
+ bad_format = True
+
+ regex_valid_char = '[_a-zA-Z0-9]'
+ name = list(name) # str -> list[char]: allows assignment at index
+ for index, char in enumerate(name):
+ if not re.fullmatch(regex_valid_char, char):
+ name[index] = '_'
+ bad_format = True
+ name = "".join(name) # list -> str
+
+ if bad_format:
+ raise InvalidPythonNameError(
+ msg="Name can only contain letters and underscores and must not commence with a numeral.",
+ invalid_name=orig_name, proposed_name=name)
+
+
+def get_valid_python_name(name: str) -> str:
+ """
+ This function validates whether the input name is a valid Python name. If the name is valid it is returned as-is.
+ If the input name is not valid, a corrected version of the name is returned.
+
+ Invalid names are names that: (Note: Corresponding auto-corrections applied are shown in brackets.)
+ - Are reserved keywords (prepend 'Obj_' to name)
+ - Start with a number (prepend underscore '_' to name)
+ - Contain any non-alpha-numeric character except the underscore (replace invalid character with underscore '_')
+
+ :param name: the name to be verified and augmented if required.
+ :returns: The original ìnput 'name', if valid, or a corrected version of the input 'name' if invalid.
+ """
+ invalid = True
+ while invalid:
+ try:
+ validate_python_name(name)
+ invalid = False
+ except InvalidPythonNameError as e:
+ log.warning('Invalid part or link name: {}. {} Changing to: {}.', name, str(e), e.proposed_name)
+ name = e.proposed_name
+ return name
+
+
+# -- Class Definitions --------------------------------------------------------------------------
+
+
+class InvalidPythonNameError(ValueError):
+ """
+ This class provides a class-specific implementation of the built-in ValueError exception.
+ It is raised when a string is determined to be an invalid Python name.
+ """
+
+ def __init__(self, msg: str, invalid_name: str = None, proposed_name: str = None, scenario_location: str = None):
+ """
+ :param msg: A message describing why the Python name is invalid.
+ :param invalid_name: The name that is the subject of the Error.
+ :param proposed_name: A proposed correction for the invalid name.
+ :param scenario_location: The scenario hierarchy location of the invalid name.
+ """
+ super(InvalidPythonNameError, self).__init__(msg)
+ self.invalid_name = invalid_name
+ self.proposed_name = proposed_name
+ self.scenario_location = scenario_location
+
+
+def get_enum_val_name(enum_obj: Enum) -> str:
+ """Get the name for an enumeration value. MyEnum.some_val returns "some_val"."""
+ return enum_obj.name
+
+
+def rel_to_timedelta(delta: relativedelta) -> timedelta:
+ """Convert a dateutil.relativedelta to a datetime.timedelta"""
+ return timedelta(days=delta.days, hours=delta.hours, minutes=delta.minutes,
+ seconds=delta.seconds, microseconds=delta.microseconds)
+
+
+def timedelta_to_rel(delta: timedelta) -> relativedelta:
+ """Convert a datetime.timedelta to a dateutil.relativedelta"""
+ # the normalized() distributes the days/seconds/micro so hours and minutes attributes have integer values
+ return relativedelta(days=delta.days, seconds=delta.seconds, microseconds=delta.microseconds).normalized()
+
+
+class UniqueIdGenerator:
+ """
+ Generates a unique id every time the static method get_new_id() is called.
+ This class is *not* multi-thread safe (there is a possibility that two threads
+ would get the same id). The start ID can be configured.
+ """
+
+ __registry = []
+
+ def __init__(self):
+ self.__next_id = 0
+ self.__registry.append(self)
+
+ def __del__(self):
+ self.__registry.remove(self)
+
+ def get_new_id(self) -> int:
+ """
+ This function returns an application-unique ID each time it is called.
+ """
+ part_id = self.__next_id
+ self.__next_id += 1
+ return part_id
+
+ @classmethod
+ def reset(cls):
+ """Reset the ID counter. This should only be used in unit tests."""
+ for gen in cls.__registry:
+ gen.__next_id = 0
+
+
+def select_object(objects: List[TAny], attr_path: str, val: TAny) -> TAny:
+ """
+ Find the object that has the given attribute equal to the given value.
+
+ :param objects: set of objects to look through
+ :param attr_path: the attribute path, in dot notation
+ :param val: the value to match
+ :return: the object, or None if none found
+
+ Example:
+
+ >>> objects = [part1, part2, part3, part4]
+ >>> obj = select_object(objects, "part_frame.name", "foo")
+ >>> assert obj.part_frame.name == "foo" # passes if there is such an object
+ >>> obj = select_object(objects, "part_frame.size", 123)
+ >>> assert obj.part_frame.width == 123 # passes if there is such an object
+ """
+ path = attr_path.split('.')
+ for obj in objects:
+ orig_obj = obj
+ for p in path:
+ obj = getattr(obj, p)
+ if obj == val:
+ return orig_obj
+ return None
+
+
+class BlockProfiler(Profile):
+ """
+ This can be used to profile various portions of Origame. Use like this:
+
+ with BlockProfiler(scenario_path):
+ ...stuff to profile...
+
+ If scenario_path is c:\\folder\path.ori, then when the "with" clauses is done, the profiling data is
+ automatically saved to c:\\folder\path.pstats and can be opened with any pstats-compatible tool
+ (PyCharm, gprof2d, etc). If out_info data was given, like this:
+
+ with BlockProfiler(scenario_path, v=1, r=2):
+ ...stuff to profile...
+
+ then the output file will be c:\\folder\path_v_1_r_2.pstats so it is easy to use the profile in multiple
+ places in one run (out_data can be an identifier for section of code).
+ """
+
+ def __init__(self, scen_path: PathType, **out_info):
+ """If out_info is not empty, appends their string rep to the stats file path name"""
+ Profile.__init__(self)
+ scen_path = Path(scen_path)
+ if out_info:
+ extra = ['{}_{}'.format(name[0], val) for name, val in out_info.items()]
+ extra_str = '_'.join(extra)
+ scen_path = scen_path.with_name(scen_path.stem + '_' + extra_str)
+ self.out_path = scen_path.with_suffix('.pstats')
+
+ def __enter__(self):
+ log.warning("Profiling starting")
+ self.enable()
+ return self
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ self.disable()
+ self.dump_stats(str(self.out_path))
+ log.warning("Profiling data saved to {}", self.out_path)
+ return False
+
+
+def ori_profile(func: TCallable, scen_path: PathType, **out_info) -> TCallable:
+ """
+ Decorator that can be used to cause every call to given function to get profiled to a .pstats file.
+
+ :param func: the function to be profiled when called
+ :param scen_path: the scenario file path
+ :param out_info: additional optional info to insert in output file path
+ :return: the callable that calls func such that it can be profiled
+
+ Note: scen_path and out_info are the same as for BlockProfiler (see class docs and its __init__ docs).
+ """
+ import functools
+
+ @functools.wraps(func)
+ def wrapper(*args, **kwargs):
+ with BlockProfiler(scen_path, **out_info):
+ result = func(*args, **kwargs)
+ return result
+
+ return wrapper
+
+
+class ClockTimer:
+ """
+ Track wall-clock time. The time stops increasing after pause(), resumes increasing
+ after resume(). The get_total_time_sec() (and associated property) only update on read.
+ """
+
+ def __init__(self, pause: bool = False):
+ """
+ Start a timer. By default, start advancing time immediately
+ :param pause: if True, do not start timing immediately, will start at next resume()
+ """
+ self._start_time_sec = time()
+ self._total_time_sec = 0
+ self._paused = pause
+
+ def reset(self, seconds: float = 0.0, pause: bool = False):
+ """
+ Reset the timer.
+ :param seconds: reset the total_time_sec to this value and, if pause=True, pause (don't track time until
+ next resume()); if pause=False, immediately resume (next call to total_time_sec is almost certain to be
+ larger than seconds)
+ :param pause: if true, pause the timer, otherwise resume (starts timing) immediately
+ """
+ self._start_time_sec = time()
+ self._total_time_sec = seconds
+ self._paused = pause
+
+ def pause(self):
+ """Pause timer. The total_time_sec() will return constant value until resume() called again."""
+ if not self._paused:
+ self._total_time_sec += (time() - self._start_time_sec)
+ self._paused = True
+
+ def resume(self):
+ """Resume the timer. The time starts counting time again."""
+ if self._paused:
+ self._start_time_sec = time()
+ self._paused = False
+
+ def get_is_paused(self) -> bool:
+ return self._paused
+
+ def get_total_time_sec(self) -> float:
+ """Get time in seconds since __init__(), not including pause periods."""
+ if self._paused:
+ return self._total_time_sec
+
+ new_time = time()
+ self._total_time_sec += (new_time - self._start_time_sec)
+ self._start_time_sec = new_time
+ return self._total_time_sec
+
+ total_time_sec = property(get_total_time_sec)
+ is_paused = property(get_is_paused)
+
+
+def plural_if(condition: Either[List[Any], bool], plural: str = 's', singular: str = '') -> str:
+ """
+ Choose between plural and singular based on a condition.
+
+ :param condition: if a boolean, True indicates pluralize, False do not. If container, pluralize if more than 1 item
+ :param plural: the letter to return if plural
+ :param singular: when singular, what to return
+ :return: plural if condition is True else return singular
+
+ Example:
+ "Found {} item{}".format(num_items, plural_if(num_items > 1))
+ "Found {} item{}".format(len(found), plural_if(found)))
+ "Found {} {}".format(len(found), plural_if(found, 'geese', 'goose')))
+ """
+ try:
+ num_nodes = len(condition)
+ condition = (num_nodes > 1)
+ except TypeError:
+ # not a container, use condition as boolean
+ condition = bool(condition)
+
+ return plural if condition else singular
+
+
+def bool_to_not(flag: bool) -> str:
+ """
+ Returns empty string if flag is False, or "not " otherwise. Example:
+
+ log.info("This flag is {}true", bool_to_not(your_flag))
+
+ will log "This flag is true" if your_flag is True, but "This flag is not true" otherwise. Note the
+ missing space between the placeholder and what will follow the "not"
+ """
+ return "" if flag else "not "
+
+
+class FileLock:
+ """
+ Context manager to lock a scenario input or output file to synchronize access by simultaneously
+ executing replications. The file name of the lock is the file to be locked suffixed with "_lock_indicator".
+ """
+
+ def __init__(self, file_name: str, max_attempts: int = 50, attempt_interval: int = 3):
+ """
+ :param file_name: The name of the file that is to be locked.
+ :param max_attempts: The max number of the attempts of acquiring a lock.
+ :param attempt_interval: The interval between two consecutive attempts. In seconds.
+ """
+ self.__path = Path(file_name + "_lock_indicator")
+ self.__max_attempts = max_attempts
+ self.__attempt_interval = attempt_interval
+ self.__file = None
+
+ def __enter__(self):
+ """
+ Locks the file given in the constructor.
+ """
+ con_scheduler = sched.scheduler()
+ locking_failed = True
+
+ def attempt_to_lock(num_attempt=0):
+ nonlocal locking_failed
+ if num_attempt < self.__max_attempts:
+ try:
+ self.__file = self.__path.open('x+')
+ locking_failed = False
+ except:
+ log.debug("Unable to lock: path={}, num_attempt={}", str(self.__path), num_attempt)
+ nxt = num_attempt + 1
+ con_scheduler.enter(self.__attempt_interval, 1, attempt_to_lock, argument=(nxt,))
+
+ con_scheduler.enter(0, 1, attempt_to_lock)
+ con_scheduler.run()
+
+ if locking_failed:
+ err_msg = "Unable to lock the file after {} attempt{}. The file: {}".format(
+ self.__max_attempts, plural_if(self.__max_attempts > 1), self.__path)
+ log.error(err_msg)
+ raise TimeoutError(err_msg)
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ """
+ Unlocks the file given in the constructor.
+ :param exc_type: (unused)
+ :param exc_val: (unused)
+ :param exc_tb: (unused)
+ """
+ try:
+ self.__file.close()
+ self.__path.unlink()
+ except:
+ log.debug("Unable to delete: path={}", str(self.__path))
+
+ def __del__(self):
+ try:
+ self.__file.close()
+ self.__path.unlink()
+ except:
+ log.debug("Unable to delete: path={}", str(self.__path))
+
+
+class IJsonable:
+ """
+ Classes that represent data structures that must be output to a JSON-compatible format can derive
+ from this class. The class provides two methods to generate a JSON-compatible representation of self,
+ and the other to set self attributes from a JSON-compatible representation of self.
+ """
+
+ def __init__(self, _unknown: Dict[str, Any], *prop_names: List[str]):
+ """
+ :param prop_names: optional, list of property names to package into and out of JSON-compatible representations
+ :param _unknown: dict of parameters not recognized (create via **unknown)
+ """
+ if _unknown:
+ msg = "These JSON fields are not recognized (likely an obsolete schema): {}"
+ fields = ', '.join(_unknown)
+ log.error(msg, fields)
+ raise ValueError(msg.format(fields))
+
+ self.__json_prop_names = prop_names
+
+ def to_json(self) -> Dict[str, Any]:
+ """
+ Return a JSON-compatible map of public attributes of self. Note: properties are not
+ considered public attributes. To include public properties, define their names in the initializer.
+ """
+ return {key: getattr(self, key) for key in self.__get_valid_keys_derived()}
+
+ def from_json(self, data: Dict[str, Any]):
+ """
+ Set the state of the derived class from the JSON data provided. Each key MUST be the name of an attribute
+ that already exists in self, either as a pure data member or as a property. A ValueError will be raised
+ otherwise.
+ """
+ valid_keys = self.__get_valid_keys_derived()
+ for key in data:
+ if key in valid_keys:
+ setattr(self, key, data[key])
+ else:
+ raise ValueError('Invalid key "{}" in JSON data', key)
+
+ def __str__(self):
+ def attr_val(key):
+ return key.capitalize().replace('_', ' '), getattr(self, key)
+
+ return '\n'.join('{}: {}'.format(*attr_val(key)) for key in sorted(self.__get_valid_keys_derived()))
+
+ def __get_valid_keys_derived(self) -> List[str]:
+ valid_keys = {attr_name for attr_name in vars(self) if not attr_name.startswith('_')}
+ valid_keys.update(self.__json_prop_names)
+ return valid_keys
+
+
+class GuardFlag:
+ """
+ Facilitates the use of "guard flags" by making them exception-safe and automatically restored, even
+ across nested calls. Guard flags are flags that are set before calling a method, so that some nested
+ action will not occur; the flag must be set back to default when method done. Without this class,
+
+ - it is a pain to ensure the flag is reset despite exception being raised by method;
+ - it is a pain to determine what value the flag should be reset to (normally, the value on entry)
+
+ Example:
+ class Foo:
+ def __init__(self):
+ self.__nested = GuardFlag(False)
+ def doABC(self):
+ # called when user clicks button, or by doAC
+ ...do A...
+ self.__doC()
+ ...do B...
+ def doAC(self):
+ # called when user selects text
+ with self.__nested(True):
+ self.doABC()
+ def __doC(self):
+ if not self.__nested:
+ do C
+ """
+
+ def __init__(self, init: bool):
+ self.__flag_value = init
+ self.__flag_on_enter = []
+
+ def set(self, value: bool):
+ self.__flag_value = value
+
+ def __call__(self, on_entry: bool):
+ self.__on_entry_value = on_entry
+ return self
+
+ def __bool__(self):
+ return self.__flag_value
+
+ def __enter__(self):
+ self.__flag_on_enter.append(self.__flag_value)
+ self.__flag_value = self.__on_entry_value
+ return self.__flag_value
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ self.__flag_value = self.__flag_on_enter.pop()
+ return False
diff --git a/origame/docs/ORIGAME Tutorial DRDC-RDDC-2018-D173.PDF b/origame/docs/ORIGAME Tutorial DRDC-RDDC-2018-D173.PDF
new file mode 100644
index 0000000..3a312d9
Binary files /dev/null and b/origame/docs/ORIGAME Tutorial DRDC-RDDC-2018-D173.PDF differ
diff --git a/origame/docs/Origame 0.7.0 beta (2017-09-13) - User Manual.pdf b/origame/docs/Origame 0.7.0 beta (2017-09-13) - User Manual.pdf
new file mode 100644
index 0000000..9f2055c
Binary files /dev/null and b/origame/docs/Origame 0.7.0 beta (2017-09-13) - User Manual.pdf differ
diff --git a/origame/docs/examples_html/example_1.html b/origame/docs/examples_html/example_1.html
new file mode 100644
index 0000000..e945af1
--- /dev/null
+++ b/origame/docs/examples_html/example_1.html
@@ -0,0 +1,13 @@
+
+
+
+
+ Origame Example 1
+
+
+
+ Home
+
+
The example 1
+
+
\ No newline at end of file
diff --git a/origame/docs/examples_html/example_2.html b/origame/docs/examples_html/example_2.html
new file mode 100644
index 0000000..b4dd6f4
--- /dev/null
+++ b/origame/docs/examples_html/example_2.html
@@ -0,0 +1,13 @@
+
+
+
+
+ Origame Example 2
+
+
+
+ Home
+
+
The example 2
+
+
\ No newline at end of file
diff --git a/origame/docs/examples_html/example_3.html b/origame/docs/examples_html/example_3.html
new file mode 100644
index 0000000..59e01f5
--- /dev/null
+++ b/origame/docs/examples_html/example_3.html
@@ -0,0 +1,13 @@
+
+
+
+
+ Origame Example 3
+
+
+
+ Home
+
+
The example 3
+
+
\ No newline at end of file
diff --git a/origame/docs/examples_html/index.html b/origame/docs/examples_html/index.html
new file mode 100644
index 0000000..96e97a3
--- /dev/null
+++ b/origame/docs/examples_html/index.html
@@ -0,0 +1,13 @@
+
+
+
+
+ Origame Examples
+
+
+
+