diff options
author | Matthias P. Braendli <matthias.braendli@mpb.li> | 2020-04-28 15:12:05 +0200 |
---|---|---|
committer | Matthias P. Braendli <matthias.braendli@mpb.li> | 2020-04-28 15:12:05 +0200 |
commit | 06a429d0148c94c35f3e4e226714299dc03af5eb (patch) | |
tree | 0cab2eaa230e6101f537c0e4d086ff5877cb800d | |
download | ODR-EDI2EDI-06a429d0148c94c35f3e4e226714299dc03af5eb.tar.gz ODR-EDI2EDI-06a429d0148c94c35f3e4e226714299dc03af5eb.tar.bz2 ODR-EDI2EDI-06a429d0148c94c35f3e4e226714299dc03af5eb.zip |
Initial commitv0.0.1
54 files changed, 10560 insertions, 0 deletions
@@ -0,0 +1,674 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/> + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + <program> Copyright (C) <year> <name of author> + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +<http://www.gnu.org/licenses/>. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +<http://www.gnu.org/philosophy/why-not-lgpl.html>. diff --git a/ChangeLog b/ChangeLog new file mode 100644 index 0000000..bdddbb6 --- /dev/null +++ b/ChangeLog @@ -0,0 +1,6 @@ +This file contains information about the changes done to +ODR-EDI2EDI in this repository + +2020-04-28: Matthias P. Braendli <matthias@mpb.li> + (v0.0.1): + Project creation diff --git a/Makefile.am b/Makefile.am new file mode 100644 index 0000000..000c5c1 --- /dev/null +++ b/Makefile.am @@ -0,0 +1,61 @@ +# Copyright (C) 2020 Matthias P. Braendli, http://opendigitalradio.org + +if IS_GIT_REPO +GITVERSION_FLAGS = -DGITVERSION="\"`git describe --dirty || echo 'unknown'`\"" +else +GITVERSION_FLAGS = +endif + +bin_PROGRAMS=odr-edi2edi + +INCLUDE=-Isrc + +common_test_CFLAGS =-Wall $(INCLUDE) $(GITVERSION_FLAGS) +common_test_CXXFLAGS =-Wall -std=c++11 $(INCLUDE) $(GITVERSION_FLAGS) +common_test_LDADD = -lpthread +common_test_SOURCES = src/main.cpp \ + lib/crc.h lib/crc.c \ + lib/ClockTAI.h lib/ClockTAI.cpp \ + lib/Globals.cpp \ + lib/Log.h lib/Log.cpp \ + lib/ReedSolomon.h lib/ReedSolomon.cpp \ + lib/RemoteControl.h \ + lib/RemoteControl.cpp \ + lib/Socket.h lib/Socket.cpp \ + lib/ThreadsafeQueue.h \ + lib/charset/charset.h lib/charset/charset.cpp \ + lib/charset/utf8.h \ + lib/charset/utf8/checked.h \ + lib/charset/utf8/core.h \ + lib/charset/utf8/unchecked.h \ + lib/edi/PFT.hpp lib/edi/PFT.cpp \ + lib/edi/STIDecoder.hpp lib/edi/STIDecoder.cpp \ + lib/edi/STIWriter.hpp lib/edi/STIWriter.cpp \ + lib/edi/ETIDecoder.hpp lib/edi/ETIDecoder.cpp \ + lib/edi/eti.hpp lib/edi/eti.cpp \ + lib/edi/buffer_unpack.hpp \ + lib/edi/common.hpp lib/edi/common.cpp \ + lib/edioutput/AFPacket.cpp \ + lib/edioutput/AFPacket.h \ + lib/edioutput/EDIConfig.h \ + lib/edioutput/Interleaver.cpp \ + lib/edioutput/Interleaver.h \ + lib/edioutput/PFT.cpp \ + lib/edioutput/PFT.h \ + lib/edioutput/TagItems.cpp \ + lib/edioutput/TagItems.h \ + lib/edioutput/TagPacket.cpp \ + lib/edioutput/TagPacket.h \ + lib/edioutput/Transport.cpp \ + lib/edioutput/Transport.h \ + lib/fec/char.h \ + lib/fec/decode_rs.h lib/fec/decode_rs_char.c \ + lib/fec/encode_rs.h lib/fec/encode_rs_char.c \ + lib/fec/fec.h \ + lib/fec/init_rs.h lib/fec/init_rs_char.c \ + lib/fec/rs-common.h + +EXTRA_DIST = COPYING NEWS README.md ChangeLog \ + lib/fec/README.md src/fec/LICENSE \ + src/charset/README + diff --git a/README.md b/README.md new file mode 100644 index 0000000..6f2f1ce --- /dev/null +++ b/README.md @@ -0,0 +1,2 @@ +ODR-EDI2EDI +=========== diff --git a/configure.ac b/configure.ac new file mode 100644 index 0000000..5f77dd8 --- /dev/null +++ b/configure.ac @@ -0,0 +1,67 @@ +# Copyright (C) 2019 Matthias P. Braendli, http://opendigitalradio.org + + +AC_PREREQ(2.69) +AC_INIT([ODR-EDI2EDI], [0.0.1], [matthias.braendli@mpb.li]) +AC_CONFIG_AUX_DIR([build-aux]) +AC_CONFIG_MACRO_DIR([m4]) +AC_CANONICAL_SYSTEM +AM_INIT_AUTOMAKE([-Wall foreign subdir-objects]) +AC_CONFIG_SRCDIR([src/main.cpp]) +AC_CONFIG_HEADER([config.h]) +AM_SILENT_RULES([yes]) + +# Checks for programs. +AC_PROG_CXX +AC_PROG_CC +AM_PROG_CC_C_O +AC_PROG_INSTALL + +AX_CXX_COMPILE_STDCXX(11,noext,mandatory) + +# Checks for libraries. +AX_PTHREAD([], AC_MSG_ERROR([requires pthread])) + +# Checks for header files. +AC_HEADER_STDC +AC_CHECK_HEADERS([arpa/inet.h fcntl.h limits.h netdb.h netinet/in.h stddef.h stdint.h stdlib.h string.h sys/ioctl.h sys/socket.h sys/time.h unistd.h]) + +AC_LANG_PUSH([C++]) +AX_CHECK_COMPILE_FLAG([-Wduplicated-cond], [CXXFLAGS="$CXXFLAGS -Wduplicated-cond"], [], ["-Werror"]) +AX_CHECK_COMPILE_FLAG([-Wduplicated-branches], [CXXFLAGS="$CXXFLAGS -Wduplicated-branches"], [], ["-Werror"]) +AX_CHECK_COMPILE_FLAG([-Wlogical-op], [CXXFLAGS="$CXXFLAGS -Wlogical-op"], [], ["-Werror"]) +AX_CHECK_COMPILE_FLAG([-Wrestrict], [CXXFLAGS="$CXXFLAGS -Wrestrict"], [], ["-Werror"]) +AX_CHECK_COMPILE_FLAG([-Wdouble-promotion], [CXXFLAGS="$CXXFLAGS -Wdouble-promotion"], [], ["-Werror"]) +AX_CHECK_COMPILE_FLAG(["-Wformat=2"], [CXXFLAGS="$CXXFLAGS -Wformat=2"], [], ["-Werror"]) + +# Linux defines MSG_NOSIGNAL, some other systems have SO_NOSIGPIPE instead +AC_MSG_CHECKING(for MSG_NOSIGNAL) +AC_COMPILE_IFELSE([ AC_LANG_PROGRAM([[ + #include <sys/socket.h> + int f = MSG_NOSIGNAL; + ]])], + [ AC_MSG_RESULT(yes) + AC_DEFINE(HAVE_MSG_NOSIGNAL, 1, [Define this symbol if you have MSG_NOSIGNAL]) ], + [ AC_MSG_RESULT(no) ]) + +AC_MSG_CHECKING(for SO_NOSIGPIPE) +AC_COMPILE_IFELSE([ AC_LANG_PROGRAM([[ + #include <sys/socket.h> + int f = SO_NOSIGPIPE; + ]])], + [ AC_MSG_RESULT(yes) + AC_DEFINE(HAVE_SO_NOSIGPIPE, 1, [Define this symbol if you have SO_NOSIGPIPE]) ], + [ AC_MSG_RESULT(no) ]) + +AC_LANG_POP([C++]) + +AM_CONDITIONAL([IS_GIT_REPO], [test -d '.git']) + +AC_CONFIG_FILES([Makefile]) +AC_OUTPUT + + + +echo "***********************************************" +echo "Configured, please run make" +echo diff --git a/lib/Globals.cpp b/lib/Globals.cpp new file mode 100644 index 0000000..6be26ec --- /dev/null +++ b/lib/Globals.cpp @@ -0,0 +1,36 @@ +/* + Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 + Her Majesty the Queen in Right of Canada (Communications Research + Center Canada) + + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +/* Ensure construction and destruction of static globals in the right order */ + +#include "Log.h" +#include "RemoteControl.h" + +// the RC needs logging, and needs to be initialised later. +Logger etiLog; +RemoteControllers rcs; + diff --git a/lib/Log.cpp b/lib/Log.cpp new file mode 100644 index 0000000..abbd69a --- /dev/null +++ b/lib/Log.cpp @@ -0,0 +1,207 @@ +/* + Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 + Her Majesty the Queen in Right of Canada (Communications Research + Center Canada) + + Copyright (C) 2018 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <list> +#include <cstdarg> +#include <cinttypes> +#include <chrono> + +#include "Log.h" + +using namespace std; + + +Logger::Logger() +{ + m_io_thread = std::thread(&Logger::io_process, this); +} + +Logger::~Logger() { + m_message_queue.trigger_wakeup(); + m_io_thread.join(); + + std::lock_guard<std::mutex> guard(m_backend_mutex); + backends.clear(); +} + +void Logger::register_backend(std::shared_ptr<LogBackend> backend) +{ + std::lock_guard<std::mutex> guard(m_backend_mutex); + backends.push_back(backend); +} + + +void Logger::log(log_level_t level, const char* fmt, ...) +{ + if (level == discard) { + return; + } + + int size = 100; + std::string str; + va_list ap; + while (1) { + str.resize(size); + va_start(ap, fmt); + int n = vsnprintf((char *)str.c_str(), size, fmt, ap); + va_end(ap); + if (n > -1 && n < size) { + str.resize(n); + break; + } + if (n > -1) + size = n + 1; + else + size *= 2; + } + + logstr(level, move(str)); +} + +void Logger::logstr(log_level_t level, std::string&& message) +{ + if (level == discard) { + return; + } + + log_message_t m(level, move(message)); + m_message_queue.push(move(m)); +} + +void Logger::io_process() +{ + while (1) { + log_message_t m; + try { + m_message_queue.wait_and_pop(m); + } + catch (const ThreadsafeQueueWakeup&) { + break; + } + + auto message = m.message; + + /* Remove a potential trailing newline. + * It doesn't look good in syslog + */ + if (message[message.length()-1] == '\n') { + message.resize(message.length()-1); + } + + { + std::lock_guard<std::mutex> guard(m_backend_mutex); + for (auto &backend : backends) { + backend->log(m.level, message); + } + + if (m.level != log_level_t::trace) { + std::cerr << levels_as_str[m.level] << " " << message << std::endl; + } + } + } +} + + +LogLine Logger::level(log_level_t level) +{ + return LogLine(this, level); +} + +LogToFile::LogToFile(const std::string& filename) : name("FILE") +{ + FILE* fd = fopen(filename.c_str(), "a"); + if (fd == nullptr) { + fprintf(stderr, "Cannot open log file !"); + throw std::runtime_error("Cannot open log file !"); + } + + log_file.reset(fd); +} + +void LogToFile::log(log_level_t level, const std::string& message) +{ + if (not (level == log_level_t::trace or level == log_level_t::discard)) { + const char* log_level_text[] = { + "DEBUG", "INFO", "WARN", "ERROR", "ALERT", "EMERG"}; + + // fprintf is thread-safe + fprintf(log_file.get(), SYSLOG_IDENT ": %s: %s\n", + log_level_text[(size_t)level], message.c_str()); + fflush(log_file.get()); + } +} + +void LogToSyslog::log(log_level_t level, const std::string& message) +{ + if (not (level == log_level_t::trace or level == log_level_t::discard)) { + int syslog_level = LOG_EMERG; + switch (level) { + case debug: syslog_level = LOG_DEBUG; break; + case info: syslog_level = LOG_INFO; break; + /* we don't have the notice level */ + case warn: syslog_level = LOG_WARNING; break; + case error: syslog_level = LOG_ERR; break; + default: syslog_level = LOG_CRIT; break; + case alert: syslog_level = LOG_ALERT; break; + case emerg: syslog_level = LOG_EMERG; break; + } + + syslog(syslog_level, SYSLOG_IDENT " %s", message.c_str()); + } +} + +LogTracer::LogTracer(const string& trace_filename) : name("TRACE") +{ + etiLog.level(info) << "Setting up TRACE to " << trace_filename; + + FILE* fd = fopen(trace_filename.c_str(), "a"); + if (fd == nullptr) { + fprintf(stderr, "Cannot open trace file !"); + throw std::runtime_error("Cannot open trace file !"); + } + m_trace_file.reset(fd); + + using namespace std::chrono; + auto now = steady_clock::now().time_since_epoch(); + m_trace_micros_startup = duration_cast<microseconds>(now).count(); + + fprintf(m_trace_file.get(), + "0,TRACER,startup at %" PRIu64 "\n", m_trace_micros_startup); +} + +void LogTracer::log(log_level_t level, const std::string& message) +{ + if (level == log_level_t::trace) { + using namespace std::chrono; + const auto now = steady_clock::now().time_since_epoch(); + const auto micros = duration_cast<microseconds>(now).count(); + + fprintf(m_trace_file.get(), "%" PRIu64 ",%s\n", + micros - m_trace_micros_startup, + message.c_str()); + } +} diff --git a/lib/Log.h b/lib/Log.h new file mode 100644 index 0000000..f20e698 --- /dev/null +++ b/lib/Log.h @@ -0,0 +1,200 @@ +/* + Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 + Her Majesty the Queen in Right of Canada (Communications Research + Center Canada) + + Copyright (C) 2018 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif + +#include <syslog.h> +#include <cstdarg> +#include <cstdio> +#include <fstream> +#include <sstream> +#include <iostream> +#include <list> +#include <stdexcept> +#include <string> +#include <map> +#include <mutex> +#include <memory> +#include <thread> +#include "ThreadsafeQueue.h" + +#define SYSLOG_IDENT PACKAGE_NAME +#define SYSLOG_FACILITY LOG_LOCAL0 + +enum log_level_t {debug = 0, info, warn, error, alert, emerg, trace, discard}; + +static const std::string levels_as_str[] = + { " ", " ", "WARN ", "ERROR", "ALERT", "EMERG", "TRACE", "-----"} ; + +/** Abstract class all backends must inherit from */ +class LogBackend { + public: + virtual ~LogBackend() {}; + virtual void log(log_level_t level, const std::string& message) = 0; + virtual std::string get_name() const = 0; +}; + +/** A Logging backend for Syslog */ +class LogToSyslog : public LogBackend { + public: + LogToSyslog() : name("SYSLOG") { + openlog(SYSLOG_IDENT, LOG_PID, SYSLOG_FACILITY); + } + + virtual ~LogToSyslog() { + closelog(); + } + + void log(log_level_t level, const std::string& message); + + std::string get_name() const { return name; } + + private: + const std::string name; + + LogToSyslog(const LogToSyslog& other) = delete; + const LogToSyslog& operator=(const LogToSyslog& other) = delete; +}; + +class LogToFile : public LogBackend { + public: + LogToFile(const std::string& filename); + void log(log_level_t level, const std::string& message); + std::string get_name() const { return name; } + + private: + const std::string name; + + struct FILEDeleter{ void operator()(FILE* fd){ if(fd) fclose(fd);}}; + std::unique_ptr<FILE, FILEDeleter> log_file; + + LogToFile(const LogToFile& other) = delete; + const LogToFile& operator=(const LogToFile& other) = delete; +}; + +class LogTracer : public LogBackend { + public: + LogTracer(const std::string& filename); + void log(log_level_t level, const std::string& message); + std::string get_name() const { return name; } + private: + std::string name; + uint64_t m_trace_micros_startup = 0; + + struct FILEDeleter{ void operator()(FILE* fd){ if(fd) fclose(fd);}}; + std::unique_ptr<FILE, FILEDeleter> m_trace_file; + + LogTracer(const LogTracer& other) = delete; + const LogTracer& operator=(const LogTracer& other) = delete; +}; + +class LogLine; + +struct log_message_t { + log_message_t(log_level_t _level, std::string&& _message) : + level(_level), + message(move(_message)) {} + + log_message_t() : + level(debug), + message("") {} + + log_level_t level; + std::string message; +}; + +class Logger { + public: + Logger(); + Logger(const Logger& other) = delete; + const Logger& operator=(const Logger& other) = delete; + ~Logger(); + + void register_backend(std::shared_ptr<LogBackend> backend); + + /* Log the message to all backends */ + void log(log_level_t level, const char* fmt, ...); + + void logstr(log_level_t level, std::string&& message); + + /* All logging IO is done in another thread */ + void io_process(void); + + /* Return a LogLine for the given level + * so that you can write etiLog.level(info) << "stuff = " << 21 */ + LogLine level(log_level_t level); + + private: + std::list<std::shared_ptr<LogBackend> > backends; + + ThreadsafeQueue<log_message_t> m_message_queue; + std::thread m_io_thread; + std::mutex m_backend_mutex; +}; + +/* etiLog is a singleton used in all parts of the program to output log messages. + * It is constructed in Globals.cpp */ +extern Logger etiLog; + +// Accumulate a line of logs, using same syntax as stringstream +// The line is logged when the LogLine gets destroyed +class LogLine { + public: + LogLine(const LogLine& logline); + const LogLine& operator=(const LogLine& other) = delete; + LogLine(Logger* logger, log_level_t level) : + logger_(logger) + { + level_ = level; + } + + // Push the new element into the stringstream + template <typename T> + LogLine& operator<<(T s) { + if (level_ != discard) { + os << s; + } + return *this; + } + + ~LogLine() + { + if (level_ != discard) { + logger_->logstr(level_, os.str()); + } + } + + private: + std::ostringstream os; + log_level_t level_; + Logger* logger_; +}; + diff --git a/lib/ReedSolomon.cpp b/lib/ReedSolomon.cpp new file mode 100644 index 0000000..1bf0b24 --- /dev/null +++ b/lib/ReedSolomon.cpp @@ -0,0 +1,118 @@ +/* + Copyright (C) 2005, 2006, 2007, 2008, 2009 Her Majesty the Queen in Right + of Canada (Communications Research Center Canada) + + Copyright (C) 2016 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This file is part of ODR-DabMux. + + ODR-DabMux is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + ODR-DabMux is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with ODR-DabMux. If not, see <http://www.gnu.org/licenses/>. + */ + +#include "ReedSolomon.h" +#include <vector> +#include <algorithm> +#include <stdexcept> +#include <sstream> +#include <stdio.h> // For galois.h ... +#include <string.h> // For memcpy + +extern "C" { +#include "fec/fec.h" +} +#include <assert.h> + +#define SYMSIZE 8 + + +ReedSolomon::ReedSolomon(int N, int K, bool reverse, int gfpoly, int firstRoot, int primElem) +{ + setReverse(reverse); + + m_N = N; + m_K = K; + + const int symsize = SYMSIZE; + const int nroots = N - K; // For EDI PFT, this must be 48 + const int pad = ((1 << symsize) - 1) - N; // is 255-N + + rsData = init_rs_char(symsize, gfpoly, firstRoot, primElem, nroots, pad); + + if (rsData == nullptr) { + std::stringstream ss; + ss << "Invalid Reed-Solomon parameters! " << + "N=" << N << " ; K=" << K << " ; pad=" << pad; + throw std::invalid_argument(ss.str()); + } +} + + +ReedSolomon::~ReedSolomon() +{ + if (rsData != nullptr) { + free_rs_char(rsData); + } +} + + +void ReedSolomon::setReverse(bool state) +{ + reverse = state; +} + + +int ReedSolomon::encode(void* data, void* fec, size_t size) +{ + uint8_t* input = reinterpret_cast<uint8_t*>(data); + uint8_t* output = reinterpret_cast<uint8_t*>(fec); + int ret = 0; + + if (reverse) { + std::vector<uint8_t> buffer(m_N); + + memcpy(&buffer[0], input, m_K); + memcpy(&buffer[m_K], output, m_N - m_K); + + ret = decode_rs_char(rsData, &buffer[0], nullptr, 0); + if ((ret != 0) && (ret != -1)) { + memcpy(input, &buffer[0], m_K); + memcpy(output, &buffer[m_K], m_N - m_K); + } + } + else { + encode_rs_char(rsData, input, output); + } + + return ret; +} + + +int ReedSolomon::encode(void* data, size_t size) +{ + uint8_t* input = reinterpret_cast<uint8_t*>(data); + int ret = 0; + + if (reverse) { + ret = decode_rs_char(rsData, input, nullptr, 0); + } + else { + encode_rs_char(rsData, input, &input[m_K]); + } + + return ret; +} diff --git a/lib/ReedSolomon.h b/lib/ReedSolomon.h new file mode 100644 index 0000000..abcef62 --- /dev/null +++ b/lib/ReedSolomon.h @@ -0,0 +1,56 @@ +/* + Copyright (C) 2005, 2006, 2007, 2008, 2009 Her Majesty the Queen in Right + of Canada (Communications Research Center Canada) + + Copyright (C) 2016 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This file is part of ODR-DabMux. + + ODR-DabMux is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + ODR-DabMux is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with ODR-DabMux. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#ifdef HAVE_CONFIG_H +# include <config.h> +#endif + +#include <stdlib.h> + +class ReedSolomon +{ +public: + ReedSolomon(int N, int K, + bool reverse = false, + int gfpoly = 0x11d, int firstRoot = 0, int primElem = 1); + ReedSolomon(const ReedSolomon& other) = delete; + ReedSolomon operator=(const ReedSolomon& other) = delete; + ~ReedSolomon(); + + void setReverse(bool state); + int encode(void* data, void* fec, size_t size); + int encode(void* data, size_t size); + +private: + int m_N; + int m_K; + + void* rsData; + bool reverse; +}; + diff --git a/lib/RemoteControl.cpp b/lib/RemoteControl.cpp new file mode 100644 index 0000000..4adb90c --- /dev/null +++ b/lib/RemoteControl.cpp @@ -0,0 +1,579 @@ +/* + Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012 + Her Majesty the Queen in Right of Canada (Communications Research + Center Canada) + + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + */ +#include <list> +#include <string> +#include <iostream> +#include <string> +#include <algorithm> + +#include "RemoteControl.h" + +using namespace std; + +RemoteControllerTelnet::~RemoteControllerTelnet() +{ + m_active = false; + + if (m_restarter_thread.joinable()) { + m_restarter_thread.join(); + } + + if (m_child_thread.joinable()) { + m_child_thread.join(); + } +} + +void RemoteControllerTelnet::restart() +{ + if (m_restarter_thread.joinable()) { + m_restarter_thread.join(); + } + + m_restarter_thread = std::thread( + &RemoteControllerTelnet::restart_thread, + this, 0); +} + +RemoteControllable::~RemoteControllable() { + rcs.remove_controllable(this); +} + +std::list<std::string> RemoteControllable::get_supported_parameters() const { + std::list<std::string> parameterlist; + for (const auto& param : m_parameters) { + parameterlist.push_back(param[0]); + } + return parameterlist; +} + +void RemoteControllers::add_controller(std::shared_ptr<BaseRemoteController> rc) { + m_controllers.push_back(rc); +} + +void RemoteControllers::enrol(RemoteControllable *rc) { + controllables.push_back(rc); +} + +void RemoteControllers::remove_controllable(RemoteControllable *rc) { + controllables.remove(rc); +} + +std::list< std::vector<std::string> > RemoteControllers::get_param_list_values(const std::string& name) { + RemoteControllable* controllable = get_controllable_(name); + + std::list< std::vector<std::string> > allparams; + for (auto ¶m : controllable->get_supported_parameters()) { + std::vector<std::string> item; + item.push_back(param); + try { + item.push_back(controllable->get_parameter(param)); + } + catch (const ParameterError &e) { + item.push_back(std::string("error: ") + e.what()); + } + + allparams.push_back(item); + } + return allparams; +} + +std::string RemoteControllers::get_param(const std::string& name, const std::string& param) { + RemoteControllable* controllable = get_controllable_(name); + return controllable->get_parameter(param); +} + +void RemoteControllers::check_faults() { + for (auto &controller : m_controllers) { + if (controller->fault_detected()) { + etiLog.level(warn) << + "Detected Remote Control fault, restarting it"; + controller->restart(); + } + } +} + +RemoteControllable* RemoteControllers::get_controllable_(const std::string& name) +{ + auto rc = std::find_if(controllables.begin(), controllables.end(), + [&](RemoteControllable* r) { return r->get_rc_name() == name; }); + + if (rc == controllables.end()) { + throw ParameterError("Module name unknown"); + } + else { + return *rc; + } +} + +void RemoteControllers::set_param( + const std::string& name, + const std::string& param, + const std::string& value) +{ + etiLog.level(info) << "RC: Setting " << name << " " << param + << " to " << value; + RemoteControllable* controllable = get_controllable_(name); + try { + return controllable->set_parameter(param, value); + } + catch (const ios_base::failure& e) { + etiLog.level(info) << "RC: Failed to set " << name << " " << param + << " to " << value << ": " << e.what(); + throw ParameterError("Cannot understand value"); + } +} + +// This runs in a separate thread, because +// it would take too long to be done in the main loop +// thread. +void RemoteControllerTelnet::restart_thread(long) +{ + m_active = false; + + if (m_child_thread.joinable()) { + m_child_thread.join(); + } + + m_child_thread = std::thread(&RemoteControllerTelnet::process, this, 0); +} + +void RemoteControllerTelnet::handle_accept(Socket::TCPSocket&& socket) +{ + const std::string welcome = PACKAGE_NAME " Remote Control CLI\n" + "Write 'help' for help.\n" + "**********\n"; + const std::string prompt = "> "; + + std::string in_message; + + try { + etiLog.level(info) << "RC: Accepted"; + + socket.sendall(welcome.data(), welcome.size()); + + while (m_active and in_message != "quit") { + socket.sendall(prompt.data(), prompt.size()); + + stringstream in_message_stream; + + char last_char = '\0'; + try { + while (last_char != '\n') { + try { + auto ret = socket.recv(&last_char, 1, 0, 1000); + if (ret == 1) { + in_message_stream << last_char; + } + else { + break; + } + } + catch (const Socket::TCPSocket::Timeout&) { + if (not m_active) { + break; + } + } + } + } + catch (const Socket::TCPSocket::Interrupted&) { + in_message_stream.clear(); + } + + + if (in_message_stream.str().size() == 0) { + etiLog.level(info) << "RC: Connection terminated"; + break; + } + + std::getline(in_message_stream, in_message); + + while (in_message.length() > 0 && + (in_message[in_message.length()-1] == '\r' || + in_message[in_message.length()-1] == '\n')) { + in_message.erase(in_message.length()-1, 1); + } + + if (in_message.length() == 0) { + continue; + } + + etiLog.level(info) << "RC: Got message '" << in_message << "'"; + + dispatch_command(socket, in_message); + } + etiLog.level(info) << "RC: Closing socket"; + socket.close(); + } + catch (const std::exception& e) { + etiLog.level(error) << "Remote control caught exception: " << e.what(); + } +} + +void RemoteControllerTelnet::process(long) +{ + try { + m_active = true; + + m_socket.listen(m_port, "localhost"); + + etiLog.level(info) << "RC: Waiting for connection on port " << m_port; + while (m_active) { + auto sock = m_socket.accept(1000); + + if (sock.valid()) { + handle_accept(move(sock)); + etiLog.level(info) << "RC: Connection closed. Waiting for connection on port " << m_port; + } + } + } + catch (const runtime_error& e) { + etiLog.level(warn) << "RC: Encountered error: " << e.what(); + } + + etiLog.level(info) << "RC: Leaving"; + m_fault = true; +} + +static std::vector<std::string> tokenise(const std::string& message) { + stringstream ss(message); + std::vector<std::string> all_tokens; + std::string item; + + while (std::getline(ss, item, ' ')) { + all_tokens.push_back(move(item)); + } + return all_tokens; +} + + +void RemoteControllerTelnet::dispatch_command(Socket::TCPSocket& socket, string command) +{ + vector<string> cmd = tokenise(command); + + if (cmd[0] == "help") { + reply(socket, + "The following commands are supported:\n" + " list\n" + " * Lists the modules that are loaded and their parameters\n" + " show MODULE\n" + " * Lists all parameters and their values from module MODULE\n" + " get MODULE PARAMETER\n" + " * Gets the value for the specified PARAMETER from module MODULE\n" + " set MODULE PARAMETER VALUE\n" + " * Sets the value for the PARAMETER ofr module MODULE\n" + " quit\n" + " * Terminate this session\n" + "\n"); + } + else if (cmd[0] == "list") { + stringstream ss; + + if (cmd.size() == 1) { + for (auto &controllable : rcs.controllables) { + ss << controllable->get_rc_name() << endl; + + list< vector<string> > params = controllable->get_parameter_descriptions(); + for (auto ¶m : params) { + ss << "\t" << param[0] << " : " << param[1] << endl; + } + } + } + else { + reply(socket, "Too many arguments for command 'list'"); + } + + reply(socket, ss.str()); + } + else if (cmd[0] == "show") { + if (cmd.size() == 2) { + try { + stringstream ss; + list< vector<string> > r = rcs.get_param_list_values(cmd[1]); + for (auto ¶m_val : r) { + ss << param_val[0] << ": " << param_val[1] << endl; + } + reply(socket, ss.str()); + + } + catch (const ParameterError &e) { + reply(socket, e.what()); + } + } + else { + reply(socket, "Incorrect parameters for command 'show'"); + } + } + else if (cmd[0] == "get") { + if (cmd.size() == 3) { + try { + string r = rcs.get_param(cmd[1], cmd[2]); + reply(socket, r); + } + catch (const ParameterError &e) { + reply(socket, e.what()); + } + } + else { + reply(socket, "Incorrect parameters for command 'get'"); + } + } + else if (cmd[0] == "set") { + if (cmd.size() >= 4) { + try { + stringstream new_param_value; + for (size_t i = 3; i < cmd.size(); i++) { + new_param_value << cmd[i]; + + if (i+1 < cmd.size()) { + new_param_value << " "; + } + } + + rcs.set_param(cmd[1], cmd[2], new_param_value.str()); + reply(socket, "ok"); + } + catch (const ParameterError &e) { + reply(socket, e.what()); + } + catch (const exception &e) { + reply(socket, "Error: Invalid parameter value. "); + } + } + else { + reply(socket, "Incorrect parameters for command 'set'"); + } + } + else if (cmd[0] == "quit") { + reply(socket, "Goodbye"); + } + else { + reply(socket, "Message not understood"); + } +} + +void RemoteControllerTelnet::reply(Socket::TCPSocket& socket, string message) +{ + stringstream ss; + ss << message << "\r\n"; + socket.sendall(message.data(), message.size()); +} + + +#if defined(HAVE_ZEROMQ) + +RemoteControllerZmq::~RemoteControllerZmq() { + m_active = false; + m_fault = false; + + if (m_restarter_thread.joinable()) { + m_restarter_thread.join(); + } + + if (m_child_thread.joinable()) { + m_child_thread.join(); + } +} + +void RemoteControllerZmq::restart() +{ + if (m_restarter_thread.joinable()) { + m_restarter_thread.join(); + } + + m_restarter_thread = std::thread(&RemoteControllerZmq::restart_thread, this); +} + +// This runs in a separate thread, because +// it would take too long to be done in the main loop +// thread. +void RemoteControllerZmq::restart_thread() +{ + m_active = false; + + if (m_child_thread.joinable()) { + m_child_thread.join(); + } + + m_child_thread = std::thread(&RemoteControllerZmq::process, this); +} + +void RemoteControllerZmq::recv_all(zmq::socket_t& pSocket, std::vector<std::string> &message) +{ + bool more = true; + do { + zmq::message_t msg; + pSocket.recv(&msg); + std::string incoming((char*)msg.data(), msg.size()); + message.push_back(incoming); + more = msg.more(); + } while (more); +} + +void RemoteControllerZmq::send_ok_reply(zmq::socket_t &pSocket) +{ + zmq::message_t msg(2); + char repCode[2] = {'o', 'k'}; + memcpy ((void*) msg.data(), repCode, 2); + pSocket.send(msg, 0); +} + +void RemoteControllerZmq::send_fail_reply(zmq::socket_t &pSocket, const std::string &error) +{ + zmq::message_t msg1(4); + char repCode[4] = {'f', 'a', 'i', 'l'}; + memcpy ((void*) msg1.data(), repCode, 4); + pSocket.send(msg1, ZMQ_SNDMORE); + + zmq::message_t msg2(error.length()); + memcpy ((void*) msg2.data(), error.c_str(), error.length()); + pSocket.send(msg2, 0); +} + +void RemoteControllerZmq::process() +{ + m_fault = false; + + // create zmq reply socket for receiving ctrl parameters + try { + zmq::socket_t repSocket(m_zmqContext, ZMQ_REP); + + // connect the socket + int hwm = 100; + int linger = 0; + repSocket.setsockopt(ZMQ_RCVHWM, &hwm, sizeof(hwm)); + repSocket.setsockopt(ZMQ_SNDHWM, &hwm, sizeof(hwm)); + repSocket.setsockopt(ZMQ_LINGER, &linger, sizeof(linger)); + repSocket.bind(m_endpoint.c_str()); + + // create pollitem that polls the ZMQ sockets + zmq::pollitem_t pollItems[] = { {repSocket, 0, ZMQ_POLLIN, 0} }; + while (m_active) { + zmq::poll(pollItems, 1, 100); + std::vector<std::string> msg; + + if (pollItems[0].revents & ZMQ_POLLIN) { + recv_all(repSocket, msg); + + std::string command((char*)msg[0].data(), msg[0].size()); + + if (msg.size() == 1 && command == "ping") { + send_ok_reply(repSocket); + } + else if (msg.size() == 1 && command == "list") { + size_t cohort_size = rcs.controllables.size(); + for (auto &controllable : rcs.controllables) { + std::stringstream ss; + ss << "{ \"name\": \"" << controllable->get_rc_name() << "\"," << + " \"params\": { "; + + list< vector<string> > params = controllable->get_parameter_descriptions(); + size_t i = 0; + for (auto ¶m : params) { + if (i > 0) { + ss << ", "; + } + + ss << "\"" << param[0] << "\": " << + "\"" << param[1] << "\""; + + i++; + } + + ss << " } }"; + + std::string msg_s = ss.str(); + + zmq::message_t zmsg(ss.str().size()); + memcpy ((void*) zmsg.data(), msg_s.data(), msg_s.size()); + + int flag = (--cohort_size > 0) ? ZMQ_SNDMORE : 0; + repSocket.send(zmsg, flag); + } + } + else if (msg.size() == 2 && command == "show") { + std::string module((char*) msg[1].data(), msg[1].size()); + try { + list< vector<string> > r = rcs.get_param_list_values(module); + size_t r_size = r.size(); + for (auto ¶m_val : r) { + std::stringstream ss; + ss << param_val[0] << ": " << param_val[1] << endl; + zmq::message_t zmsg(ss.str().size()); + memcpy(zmsg.data(), ss.str().data(), ss.str().size()); + + int flag = (--r_size > 0) ? ZMQ_SNDMORE : 0; + repSocket.send(zmsg, flag); + } + } + catch (const ParameterError &err) { + send_fail_reply(repSocket, err.what()); + } + } + else if (msg.size() == 3 && command == "get") { + std::string module((char*) msg[1].data(), msg[1].size()); + std::string parameter((char*) msg[2].data(), msg[2].size()); + + try { + std::string value = rcs.get_param(module, parameter); + zmq::message_t zmsg(value.size()); + memcpy ((void*) zmsg.data(), value.data(), value.size()); + repSocket.send(zmsg, 0); + } + catch (const ParameterError &err) { + send_fail_reply(repSocket, err.what()); + } + } + else if (msg.size() == 4 && command == "set") { + std::string module((char*) msg[1].data(), msg[1].size()); + std::string parameter((char*) msg[2].data(), msg[2].size()); + std::string value((char*) msg[3].data(), msg[3].size()); + + try { + rcs.set_param(module, parameter, value); + send_ok_reply(repSocket); + } + catch (const ParameterError &err) { + send_fail_reply(repSocket, err.what()); + } + } + else { + send_fail_reply(repSocket, + "Unsupported command. commands: list, show, get, set"); + } + } + } + repSocket.close(); + } + catch (const zmq::error_t &e) { + etiLog.level(error) << "ZMQ RC error: " << std::string(e.what()); + } + catch (const std::exception& e) { + etiLog.level(error) << "ZMQ RC caught exception: " << e.what(); + m_fault = true; + } +} + +#endif + diff --git a/lib/RemoteControl.h b/lib/RemoteControl.h new file mode 100644 index 0000000..2358b3a --- /dev/null +++ b/lib/RemoteControl.h @@ -0,0 +1,251 @@ +/* + Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012 + Her Majesty the Queen in Right of Canada (Communications Research + Center Canada) + + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + This module adds remote-control capability to some of the dabmux/dabmod modules. + */ +/* + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + */ + +#pragma once + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif + +#if defined(HAVE_ZEROMQ) +# include "zmq.hpp" +#endif + +#include <list> +#include <map> +#include <memory> +#include <string> +#include <atomic> +#include <iostream> +#include <thread> +#include <stdexcept> + +#include "Log.h" +#include "Socket.h" + +#define RC_ADD_PARAMETER(p, desc) { \ + std::vector<std::string> p; \ + p.push_back(#p); \ + p.push_back(desc); \ + m_parameters.push_back(p); \ +} + +class ParameterError : public std::exception +{ + public: + ParameterError(std::string message) : m_message(message) {} + ~ParameterError() throw() {} + const char* what() const throw() { return m_message.c_str(); } + + private: + std::string m_message; +}; + +class RemoteControllable; + +/* Remote controllers (that recieve orders from the user) + * must implement BaseRemoteController + */ +class BaseRemoteController { + public: + /* When this returns one, the remote controller cannot be + * used anymore, and must be restarted + */ + virtual bool fault_detected() = 0; + + /* In case of a fault, the remote controller can be + * restarted. + */ + virtual void restart() = 0; + + virtual ~BaseRemoteController() {} +}; + +/* Objects that support remote control must implement the following class */ +class RemoteControllable { + public: + RemoteControllable(const std::string& name) : + m_rc_name(name) {} + + RemoteControllable(const RemoteControllable& other) = delete; + RemoteControllable& operator=(const RemoteControllable& other) = delete; + + virtual ~RemoteControllable(); + + /* return a short name used to identify the controllable. + * It might be used in the commands the user has to type, so keep + * it short + */ + virtual std::string get_rc_name() const { return m_rc_name; } + + /* Return a list of possible parameters that can be set */ + virtual std::list<std::string> get_supported_parameters() const; + + /* Return a mapping of the descriptions of all parameters */ + virtual std::list< std::vector<std::string> > + get_parameter_descriptions() const + { + return m_parameters; + } + + /* Base function to set parameters. */ + virtual void set_parameter( + const std::string& parameter, + const std::string& value) = 0; + + /* Getting a parameter always returns a string. */ + virtual const std::string get_parameter(const std::string& parameter) const = 0; + + protected: + std::string m_rc_name; + std::list< std::vector<std::string> > m_parameters; +}; + +/* Holds all our remote controllers and controlled object. + */ +class RemoteControllers { + public: + void add_controller(std::shared_ptr<BaseRemoteController> rc); + void enrol(RemoteControllable *rc); + void remove_controllable(RemoteControllable *rc); + void check_faults(); + std::list< std::vector<std::string> > get_param_list_values(const std::string& name); + std::string get_param(const std::string& name, const std::string& param); + + void set_param( + const std::string& name, + const std::string& param, + const std::string& value); + + std::list<RemoteControllable*> controllables; + + private: + RemoteControllable* get_controllable_(const std::string& name); + + std::list<std::shared_ptr<BaseRemoteController> > m_controllers; +}; + +/* rcs is a singleton used in all parts of the program to interact with the RC. + * It is constructed in Globals.cpp */ +extern RemoteControllers rcs; + +/* Implements a Remote controller based on a simple telnet CLI + * that listens on localhost + */ +class RemoteControllerTelnet : public BaseRemoteController { + public: + RemoteControllerTelnet() + : m_active(false), + m_fault(false), + m_port(0) { } + + RemoteControllerTelnet(int port) + : m_active(port > 0), + m_fault(false), + m_port(port) + { + restart(); + } + + + RemoteControllerTelnet& operator=(const RemoteControllerTelnet& other) = delete; + RemoteControllerTelnet(const RemoteControllerTelnet& other) = delete; + + ~RemoteControllerTelnet(); + + virtual bool fault_detected() { return m_fault; } + + virtual void restart(); + + private: + void restart_thread(long); + + void process(long); + + void dispatch_command(Socket::TCPSocket& socket, std::string command); + void reply(Socket::TCPSocket& socket, std::string message); + void handle_accept(Socket::TCPSocket&& socket); + + std::atomic<bool> m_active; + + /* This is set to true if a fault occurred */ + std::atomic<bool> m_fault; + std::thread m_restarter_thread; + + std::thread m_child_thread; + + Socket::TCPSocket m_socket; + int m_port; +}; + +#if defined(HAVE_ZEROMQ) +/* Implements a Remote controller using ZMQ transportlayer + * that listens on localhost + */ +class RemoteControllerZmq : public BaseRemoteController { + public: + RemoteControllerZmq() + : m_active(false), m_fault(false), + m_zmqContext(1), + m_endpoint("") { } + + RemoteControllerZmq(const std::string& endpoint) + : m_active(not endpoint.empty()), m_fault(false), + m_zmqContext(1), + m_endpoint(endpoint), + m_child_thread(&RemoteControllerZmq::process, this) { } + + RemoteControllerZmq& operator=(const RemoteControllerZmq& other) = delete; + RemoteControllerZmq(const RemoteControllerZmq& other) = delete; + + ~RemoteControllerZmq(); + + virtual bool fault_detected() { return m_fault; } + + virtual void restart(); + + private: + void restart_thread(); + + void recv_all(zmq::socket_t &pSocket, std::vector<std::string> &message); + void send_ok_reply(zmq::socket_t &pSocket); + void send_fail_reply(zmq::socket_t &pSocket, const std::string &error); + void process(); + + std::atomic<bool> m_active; + + /* This is set to true if a fault occurred */ + std::atomic<bool> m_fault; + std::thread m_restarter_thread; + + zmq::context_t m_zmqContext; + + std::string m_endpoint; + std::thread m_child_thread; +}; +#endif + diff --git a/lib/Socket.cpp b/lib/Socket.cpp new file mode 100644 index 0000000..159de7e --- /dev/null +++ b/lib/Socket.cpp @@ -0,0 +1,972 @@ +/* + Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009 Her Majesty the + Queen in Right of Canada (Communications Research Center Canada) + + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. +*/ + +#include "Socket.h" + +#include <iostream> +#include <cstdio> +#include <cstring> +#include <cerrno> +#include <fcntl.h> +#include <poll.h> + +namespace Socket { + +using namespace std; + +void InetAddress::resolveUdpDestination(const std::string& destination, int port) +{ + char service[NI_MAXSERV]; + snprintf(service, NI_MAXSERV-1, "%d", port); + + struct addrinfo hints; + memset(&hints, 0, sizeof(struct addrinfo)); + hints.ai_family = AF_INET; + hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ + hints.ai_flags = 0; + hints.ai_protocol = 0; + + struct addrinfo *result, *rp; + int s = getaddrinfo(destination.c_str(), service, &hints, &result); + if (s != 0) { + throw runtime_error(string("getaddrinfo failed: ") + gai_strerror(s)); + } + + for (rp = result; rp != nullptr; rp = rp->ai_next) { + // Take the first result + memcpy(&addr, rp->ai_addr, rp->ai_addrlen); + break; + } + + freeaddrinfo(result); + + if (rp == nullptr) { + throw runtime_error("Could not resolve"); + } +} + +UDPPacket::UDPPacket() { } + +UDPPacket::UDPPacket(size_t initSize) : + buffer(initSize), + address() +{ } + + +UDPSocket::UDPSocket() : + m_sock(INVALID_SOCKET) +{ + reinit(0, ""); +} + +UDPSocket::UDPSocket(int port) : + m_sock(INVALID_SOCKET) +{ + reinit(port, ""); +} + +UDPSocket::UDPSocket(int port, const std::string& name) : + m_sock(INVALID_SOCKET) +{ + reinit(port, name); +} + + +void UDPSocket::setBlocking(bool block) +{ + int res = fcntl(m_sock, F_SETFL, block ? 0 : O_NONBLOCK); + if (res == -1) { + throw runtime_error(string("Can't change blocking state of socket: ") + strerror(errno)); + } +} + +void UDPSocket::reinit(int port) +{ + return reinit(port, ""); +} + +void UDPSocket::reinit(int port, const std::string& name) +{ + if (m_sock != INVALID_SOCKET) { + ::close(m_sock); + } + + if (port == 0) { + // No need to bind to a given port, creating the + // socket is enough + m_sock = ::socket(AF_INET, SOCK_DGRAM, 0); + return; + } + + char service[NI_MAXSERV]; + snprintf(service, NI_MAXSERV-1, "%d", port); + + struct addrinfo hints; + memset(&hints, 0, sizeof(struct addrinfo)); + hints.ai_family = AF_INET; + hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ + hints.ai_flags = AI_PASSIVE; /* For wildcard IP address */ + hints.ai_protocol = 0; /* Any protocol */ + hints.ai_canonname = nullptr; + hints.ai_addr = nullptr; + hints.ai_next = nullptr; + + struct addrinfo *result, *rp; + int s = getaddrinfo(name.empty() ? nullptr : name.c_str(), + port == 0 ? nullptr : service, + &hints, &result); + if (s != 0) { + throw runtime_error(string("getaddrinfo failed: ") + gai_strerror(s)); + } + + /* getaddrinfo() returns a list of address structures. + Try each address until we successfully bind(2). + If socket(2) (or bind(2)) fails, we (close the socket + and) try the next address. */ + for (rp = result; rp != nullptr; rp = rp->ai_next) { + int sfd = ::socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); + if (sfd == -1) { + continue; + } + + if (::bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) { + m_sock = sfd; + break; + } + + ::close(sfd); + } + + freeaddrinfo(result); + + if (rp == nullptr) { + throw runtime_error("Could not bind"); + } +} + +void UDPSocket::close() +{ + if (m_sock != INVALID_SOCKET) { + ::close(m_sock); + } + + m_sock = INVALID_SOCKET; +} + +UDPSocket::~UDPSocket() +{ + if (m_sock != INVALID_SOCKET) { + ::close(m_sock); + } +} + + +UDPPacket UDPSocket::receive(size_t max_size) +{ + UDPPacket packet(max_size); + socklen_t addrSize; + addrSize = sizeof(*packet.address.as_sockaddr()); + ssize_t ret = recvfrom(m_sock, + packet.buffer.data(), + packet.buffer.size(), + 0, + packet.address.as_sockaddr(), + &addrSize); + + if (ret == SOCKET_ERROR) { + packet.buffer.resize(0); + + // This suppresses the -Wlogical-op warning +#if EAGAIN == EWOULDBLOCK + if (errno == EAGAIN) +#else + if (errno == EAGAIN or errno == EWOULDBLOCK) +#endif + { + return 0; + } + throw runtime_error(string("Can't receive data: ") + strerror(errno)); + } + + packet.buffer.resize(ret); + return packet; +} + +void UDPSocket::send(UDPPacket& packet) +{ + const int ret = sendto(m_sock, packet.buffer.data(), packet.buffer.size(), 0, + packet.address.as_sockaddr(), sizeof(*packet.address.as_sockaddr())); + if (ret == SOCKET_ERROR && errno != ECONNREFUSED) { + throw runtime_error(string("Can't send UDP packet: ") + strerror(errno)); + } +} + + +void UDPSocket::send(const std::vector<uint8_t>& data, InetAddress destination) +{ + const int ret = sendto(m_sock, data.data(), data.size(), 0, + destination.as_sockaddr(), sizeof(*destination.as_sockaddr())); + if (ret == SOCKET_ERROR && errno != ECONNREFUSED) { + throw runtime_error(string("Can't send UDP packet: ") + strerror(errno)); + } +} + +void UDPSocket::joinGroup(const char* groupname, const char* if_addr) +{ + ip_mreqn group; + if ((group.imr_multiaddr.s_addr = inet_addr(groupname)) == INADDR_NONE) { + throw runtime_error("Cannot convert multicast group name"); + } + if (!IN_MULTICAST(ntohl(group.imr_multiaddr.s_addr))) { + throw runtime_error("Group name is not a multicast address"); + } + + if (if_addr) { + group.imr_address.s_addr = inet_addr(if_addr); + } + else { + group.imr_address.s_addr = htons(INADDR_ANY); + } + group.imr_ifindex = 0; + if (setsockopt(m_sock, IPPROTO_IP, IP_ADD_MEMBERSHIP, &group, sizeof(group)) + == SOCKET_ERROR) { + throw runtime_error(string("Can't join multicast group") + strerror(errno)); + } +} + +void UDPSocket::setMulticastSource(const char* source_addr) +{ + struct in_addr addr; + if (inet_aton(source_addr, &addr) == 0) { + throw runtime_error(string("Can't parse source address") + strerror(errno)); + } + + if (setsockopt(m_sock, IPPROTO_IP, IP_MULTICAST_IF, &addr, sizeof(addr)) + == SOCKET_ERROR) { + throw runtime_error(string("Can't set source address") + strerror(errno)); + } +} + +void UDPSocket::setMulticastTTL(int ttl) +{ + if (setsockopt(m_sock, IPPROTO_IP, IP_MULTICAST_TTL, &ttl, sizeof(ttl)) + == SOCKET_ERROR) { + throw runtime_error(string("Can't set multicast ttl") + strerror(errno)); + } +} + +UDPReceiver::UDPReceiver() { } + +UDPReceiver::~UDPReceiver() { + m_stop = true; + m_sock.close(); + if (m_thread.joinable()) { + m_thread.join(); + } +} + +void UDPReceiver::start(int port, const string& bindto, const string& mcastaddr, size_t max_packets_queued) { + m_port = port; + m_bindto = bindto; + m_mcastaddr = mcastaddr; + m_max_packets_queued = max_packets_queued; + m_thread = std::thread(&UDPReceiver::m_run, this); +} + +std::vector<uint8_t> UDPReceiver::get_packet_buffer() +{ + if (m_stop) { + throw runtime_error("UDP Receiver not running"); + } + + UDPPacket p; + m_packets.wait_and_pop(p); + + return p.buffer; +} + +void UDPReceiver::m_run() +{ + // Ensure that stop is set to true in case of exception or return + struct SetStopOnDestruct { + SetStopOnDestruct(atomic<bool>& stop) : m_stop(stop) {} + ~SetStopOnDestruct() { m_stop = true; } + private: atomic<bool>& m_stop; + } autoSetStop(m_stop); + + if (IN_MULTICAST(ntohl(inet_addr(m_mcastaddr.c_str())))) { + m_sock.reinit(m_port, m_mcastaddr); + m_sock.setMulticastSource(m_bindto.c_str()); + m_sock.joinGroup(m_mcastaddr.c_str(), m_bindto.c_str()); + } + else { + m_sock.reinit(m_port, m_bindto); + } + + while (not m_stop) { + constexpr size_t packsize = 8192; + try { + auto packet = m_sock.receive(packsize); + if (packet.buffer.size() == packsize) { + // TODO replace fprintf + fprintf(stderr, "Warning, possible UDP truncation\n"); + } + + // If this blocks, the UDP socket will lose incoming packets + m_packets.push_wait_if_full(packet, m_max_packets_queued); + } + catch (const std::runtime_error& e) { + // TODO replace fprintf + // TODO handle intr + fprintf(stderr, "Socket error: %s\n", e.what()); + m_stop = true; + } + } +} + + +TCPSocket::TCPSocket() +{ +} + +TCPSocket::~TCPSocket() +{ + if (m_sock != -1) { + ::close(m_sock); + } +} + +TCPSocket::TCPSocket(TCPSocket&& other) : + m_sock(other.m_sock), + m_remote_address(move(other.m_remote_address)) +{ + if (other.m_sock != -1) { + other.m_sock = -1; + } +} + +TCPSocket& TCPSocket::operator=(TCPSocket&& other) +{ + swap(m_remote_address, other.m_remote_address); + + m_sock = other.m_sock; + if (other.m_sock != -1) { + other.m_sock = -1; + } + + return *this; +} + +bool TCPSocket::valid() const +{ + return m_sock != -1; +} + +void TCPSocket::connect(const std::string& hostname, int port, bool nonblock) +{ + if (m_sock != INVALID_SOCKET) { + throw std::logic_error("You may only connect an invalid TCPSocket"); + } + + char service[NI_MAXSERV]; + snprintf(service, NI_MAXSERV-1, "%d", port); + + /* Obtain address(es) matching host/port */ + struct addrinfo hints; + memset(&hints, 0, sizeof(struct addrinfo)); + hints.ai_family = AF_INET; + hints.ai_socktype = SOCK_STREAM; + hints.ai_flags = 0; + hints.ai_protocol = 0; + + struct addrinfo *result, *rp; + int s = getaddrinfo(hostname.c_str(), service, &hints, &result); + if (s != 0) { + throw runtime_error(string("getaddrinfo failed: ") + gai_strerror(s)); + } + + /* getaddrinfo() returns a list of address structures. + Try each address until we successfully connect(2). + If socket(2) (or connect(2)) fails, we (close the socket + and) try the next address. */ + + for (rp = result; rp != nullptr; rp = rp->ai_next) { + int sfd = ::socket(rp->ai_family, rp->ai_socktype, + rp->ai_protocol); + if (sfd == -1) + continue; + + if (nonblock) { + int flags = fcntl(sfd, F_GETFL); + if (flags == -1) { + std::string errstr(strerror(errno)); + throw std::runtime_error("TCP: Could not get socket flags: " + errstr); + } + + if (fcntl(sfd, F_SETFL, flags | O_NONBLOCK) == -1) { + std::string errstr(strerror(errno)); + throw std::runtime_error("TCP: Could not set O_NONBLOCK: " + errstr); + } + } + + int ret = ::connect(sfd, rp->ai_addr, rp->ai_addrlen); + if (ret != -1 or (ret == -1 and errno == EINPROGRESS)) { + m_sock = sfd; + break; + } + + ::close(sfd); + } + + if (m_sock != INVALID_SOCKET) { +#if defined(HAVE_SO_NOSIGPIPE) + int val = 1; + if (setsockopt(m_sock, SOL_SOCKET, SO_NOSIGPIPE, &val, sizeof(val)) + == SOCKET_ERROR) { + throw std::runtime_error("Can't set SO_NOSIGPIPE"); + } +#endif + } + + freeaddrinfo(result); /* No longer needed */ + + if (rp == nullptr) { + throw runtime_error("Could not connect"); + } + +} + +void TCPSocket::listen(int port, const string& name) +{ + if (m_sock != INVALID_SOCKET) { + throw std::logic_error("You may only listen with an invalid TCPSocket"); + } + + char service[NI_MAXSERV]; + snprintf(service, NI_MAXSERV-1, "%d", port); + + struct addrinfo hints; + memset(&hints, 0, sizeof(struct addrinfo)); + hints.ai_family = AF_INET; + hints.ai_socktype = SOCK_STREAM; + hints.ai_flags = AI_PASSIVE; /* For wildcard IP address */ + hints.ai_protocol = 0; + hints.ai_canonname = nullptr; + hints.ai_addr = nullptr; + hints.ai_next = nullptr; + + struct addrinfo *result, *rp; + int s = getaddrinfo(name.empty() ? nullptr : name.c_str(), service, &hints, &result); + if (s != 0) { + throw runtime_error(string("getaddrinfo failed: ") + gai_strerror(s)); + } + + /* getaddrinfo() returns a list of address structures. + Try each address until we successfully bind(2). + If socket(2) (or bind(2)) fails, we (close the socket + and) try the next address. */ + for (rp = result; rp != nullptr; rp = rp->ai_next) { + int sfd = ::socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); + if (sfd == -1) { + continue; + } + + int reuse_setting = 1; + if (setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR, &reuse_setting, sizeof(reuse_setting)) == -1) { + throw runtime_error("Can't reuse address"); + } + + if (::bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) { + m_sock = sfd; + break; + } + + ::close(sfd); + } + + freeaddrinfo(result); + + if (m_sock != INVALID_SOCKET) { +#if defined(HAVE_SO_NOSIGPIPE) + int val = 1; + if (setsockopt(m_sock, SOL_SOCKET, SO_NOSIGPIPE, + &val, sizeof(val)) < 0) { + throw std::runtime_error("Can't set SO_NOSIGPIPE"); + } +#endif + + int ret = ::listen(m_sock, 0); + if (ret == -1) { + throw std::runtime_error(string("Could not listen: ") + strerror(errno)); + } + } + + if (rp == nullptr) { + throw runtime_error("Could not bind"); + } +} + +void TCPSocket::close() +{ + ::close(m_sock); + m_sock = -1; +} + +TCPSocket TCPSocket::accept(int timeout_ms) +{ + if (timeout_ms == 0) { + InetAddress remote_addr; + socklen_t client_len = sizeof(remote_addr.addr); + int sockfd = ::accept(m_sock, remote_addr.as_sockaddr(), &client_len); + TCPSocket s(sockfd, remote_addr); + return s; + } + else { + struct pollfd fds[1]; + fds[0].fd = m_sock; + fds[0].events = POLLIN; + + int retval = poll(fds, 1, timeout_ms); + + if (retval == -1) { + std::string errstr(strerror(errno)); + throw std::runtime_error("TCP Socket accept error: " + errstr); + } + else if (retval > 0) { + InetAddress remote_addr; + socklen_t client_len = sizeof(remote_addr.addr); + int sockfd = ::accept(m_sock, remote_addr.as_sockaddr(), &client_len); + TCPSocket s(sockfd, remote_addr); + return s; + } + else { + TCPSocket s(-1); + return s; + } + } +} + +ssize_t TCPSocket::sendall(const void *buffer, size_t buflen) +{ + uint8_t *buf = (uint8_t*)buffer; + while (buflen > 0) { + /* On Linux, the MSG_NOSIGNAL flag ensures that the process + * would not receive a SIGPIPE and die. + * Other systems have SO_NOSIGPIPE set on the socket for the + * same effect. */ +#if defined(HAVE_MSG_NOSIGNAL) + const int flags = MSG_NOSIGNAL; +#else + const int flags = 0; +#endif + ssize_t sent = ::send(m_sock, buf, buflen, flags); + if (sent < 0) { + return -1; + } + else { + buf += sent; + buflen -= sent; + } + } + return buflen; +} + +ssize_t TCPSocket::send(const void* data, size_t size, int timeout_ms) +{ + if (timeout_ms) { + struct pollfd fds[1]; + fds[0].fd = m_sock; + fds[0].events = POLLOUT; + + const int retval = poll(fds, 1, timeout_ms); + + if (retval == -1) { + throw std::runtime_error(string("TCP Socket send error on poll(): ") + strerror(errno)); + } + else if (retval == 0) { + // Timed out + return 0; + } + } + + /* On Linux, the MSG_NOSIGNAL flag ensures that the process would not + * receive a SIGPIPE and die. + * Other systems have SO_NOSIGPIPE set on the socket for the same effect. */ +#if defined(HAVE_MSG_NOSIGNAL) + const int flags = MSG_NOSIGNAL; +#else + const int flags = 0; +#endif + const ssize_t ret = ::send(m_sock, (const char*)data, size, flags); + + if (ret == SOCKET_ERROR) { + throw std::runtime_error(string("TCP Socket send error: ") + strerror(errno)); + } + return ret; +} + +ssize_t TCPSocket::recv(void *buffer, size_t length, int flags) +{ + ssize_t ret = ::recv(m_sock, buffer, length, flags); + if (ret == -1) { + std::string errstr(strerror(errno)); + throw std::runtime_error("TCP receive error: " + errstr); + } + return ret; +} + +ssize_t TCPSocket::recv(void *buffer, size_t length, int flags, int timeout_ms) +{ + struct pollfd fds[1]; + fds[0].fd = m_sock; + fds[0].events = POLLIN; + + int retval = poll(fds, 1, timeout_ms); + + if (retval == -1 and errno == EINTR) { + throw Interrupted(); + } + else if (retval == -1) { + std::string errstr(strerror(errno)); + throw std::runtime_error("TCP receive with poll() error: " + errstr); + } + else if (retval > 0 and (fds[0].revents & POLLIN)) { + ssize_t ret = ::recv(m_sock, buffer, length, flags); + if (ret == -1) { + if (errno == ECONNREFUSED) { + return 0; + } + std::string errstr(strerror(errno)); + throw std::runtime_error("TCP receive after poll() error: " + errstr); + } + return ret; + } + else { + throw Timeout(); + } +} + +TCPSocket::TCPSocket(int sockfd) : + m_sock(sockfd), + m_remote_address() +{ } + +TCPSocket::TCPSocket(int sockfd, InetAddress remote_address) : + m_sock(sockfd), + m_remote_address(remote_address) +{ } + +void TCPClient::connect(const std::string& hostname, int port) +{ + m_hostname = hostname; + m_port = port; + reconnect(); +} + +ssize_t TCPClient::recv(void *buffer, size_t length, int flags, int timeout_ms) +{ + try { + ssize_t ret = m_sock.recv(buffer, length, flags, timeout_ms); + + if (ret == 0) { + m_sock.close(); + reconnect(); + } + + return ret; + } + catch (const TCPSocket::Interrupted&) { + return -1; + } + catch (const TCPSocket::Timeout&) { + return 0; + } + + return 0; +} + +void TCPClient::reconnect() +{ + TCPSocket newsock; + m_sock = std::move(newsock); + m_sock.connect(m_hostname, m_port, true); +} + +TCPConnection::TCPConnection(TCPSocket&& sock) : + queue(), + m_running(true), + m_sender_thread(), + m_sock(move(sock)) +{ +#if MISSING_OWN_ADDR + auto own_addr = m_sock.getOwnAddress(); + auto addr = m_sock.getRemoteAddress(); + etiLog.level(debug) << "New TCP Connection on port " << + own_addr.getPort() << " from " << + addr.getHostAddress() << ":" << addr.getPort(); +#endif + m_sender_thread = std::thread(&TCPConnection::process, this); +} + +TCPConnection::~TCPConnection() +{ + m_running = false; + vector<uint8_t> termination_marker; + queue.push(termination_marker); + if (m_sender_thread.joinable()) { + m_sender_thread.join(); + } +} + +void TCPConnection::process() +{ + while (m_running) { + vector<uint8_t> data; + queue.wait_and_pop(data); + + if (data.empty()) { + // empty vector is the termination marker + m_running = false; + break; + } + + try { + ssize_t remaining = data.size(); + const uint8_t *buf = reinterpret_cast<const uint8_t*>(data.data()); + const int timeout_ms = 10; // Less than one ETI frame + + while (m_running and remaining > 0) { + const ssize_t sent = m_sock.send(buf, remaining, timeout_ms); + if (sent < 0 or sent > remaining) { + throw std::logic_error("Invalid TCPSocket::send() return value"); + } + remaining -= sent; + buf += sent; + } + } + catch (const std::runtime_error& e) { + m_running = false; + } + } + +#if MISSING_OWN_ADDR + auto own_addr = m_sock.getOwnAddress(); + auto addr = m_sock.getRemoteAddress(); + etiLog.level(debug) << "Dropping TCP Connection on port " << + own_addr.getPort() << " from " << + addr.getHostAddress() << ":" << addr.getPort(); +#endif +} + + +TCPDataDispatcher::TCPDataDispatcher(size_t max_queue_size) : + m_max_queue_size(max_queue_size) +{ +} + +TCPDataDispatcher::~TCPDataDispatcher() +{ + m_running = false; + m_connections.clear(); + m_listener_socket.close(); + if (m_listener_thread.joinable()) { + m_listener_thread.join(); + } +} + +void TCPDataDispatcher::start(int port, const string& address) +{ + m_listener_socket.listen(port, address); + + m_running = true; + m_listener_thread = std::thread(&TCPDataDispatcher::process, this); +} + +void TCPDataDispatcher::write(const vector<uint8_t>& data) +{ + if (not m_running) { + throw runtime_error(m_exception_data); + } + + for (auto& connection : m_connections) { + connection.queue.push(data); + } + + m_connections.remove_if( + [&](const TCPConnection& conn){ return conn.queue.size() > m_max_queue_size; }); +} + +void TCPDataDispatcher::process() +{ + try { + const int timeout_ms = 1000; + + while (m_running) { + // Add a new TCPConnection to the list, constructing it from the client socket + auto sock = m_listener_socket.accept(timeout_ms); + if (sock.valid()) { + m_connections.emplace(m_connections.begin(), move(sock)); + } + } + } + catch (const std::runtime_error& e) { + m_exception_data = string("TCPDataDispatcher error: ") + e.what(); + m_running = false; + } +} + +TCPReceiveServer::TCPReceiveServer(size_t blocksize) : + m_blocksize(blocksize) +{ +} + +void TCPReceiveServer::start(int listen_port, const std::string& address) +{ + m_listener_socket.listen(listen_port, address); + + m_running = true; + m_listener_thread = std::thread(&TCPReceiveServer::process, this); +} + +TCPReceiveServer::~TCPReceiveServer() +{ + m_running = false; + if (m_listener_thread.joinable()) { + m_listener_thread.join(); + } +} + +vector<uint8_t> TCPReceiveServer::receive() +{ + vector<uint8_t> buffer; + m_queue.try_pop(buffer); + + // we can ignore try_pop()'s return value, because + // if it is unsuccessful the buffer is not touched. + return buffer; +} + +void TCPReceiveServer::process() +{ + constexpr int timeout_ms = 1000; + constexpr int disconnect_timeout_ms = 10000; + constexpr int max_num_timeouts = disconnect_timeout_ms / timeout_ms; + + while (m_running) { + auto sock = m_listener_socket.accept(timeout_ms); + + int num_timeouts = 0; + + while (m_running and sock.valid()) { + try { + vector<uint8_t> buf(m_blocksize); + ssize_t r = sock.recv(buf.data(), buf.size(), 0, timeout_ms); + if (r < 0) { + throw logic_error("Invalid recv return value"); + } + else if (r == 0) { + sock.close(); + break; + } + else { + buf.resize(r); + m_queue.push(move(buf)); + } + } + catch (const TCPSocket::Interrupted&) { + break; + } + catch (const TCPSocket::Timeout&) { + num_timeouts++; + } + + if (num_timeouts > max_num_timeouts) { + sock.close(); + } + } + } +} + +TCPSendClient::TCPSendClient(const std::string& hostname, int port) : + m_hostname(hostname), + m_port(port), + m_running(true) +{ + m_sender_thread = std::thread(&TCPSendClient::process, this); +} + +TCPSendClient::~TCPSendClient() +{ + m_running = false; + m_queue.trigger_wakeup(); + if (m_sender_thread.joinable()) { + m_sender_thread.join(); + } +} + +void TCPSendClient::sendall(const std::vector<uint8_t>& buffer) +{ + if (not m_running) { + throw runtime_error(m_exception_data); + } + + m_queue.push(buffer); +} + +void TCPSendClient::process() +{ + try { + while (m_running) { + if (m_is_connected) { + try { + vector<uint8_t> incoming; + m_queue.wait_and_pop(incoming); + if (m_sock.sendall(incoming.data(), incoming.size()) == -1) { + m_is_connected = false; + m_sock = TCPSocket(); + } + } + catch (const ThreadsafeQueueWakeup&) { + break; + } + } + else { + try { + m_sock.connect(m_hostname, m_port); + m_is_connected = true; + } + catch (const runtime_error& e) { + m_is_connected = false; + this_thread::sleep_for(chrono::seconds(1)); + } + } + } + } + catch (const runtime_error& e) { + m_exception_data = e.what(); + m_running = false; + } +} + +} diff --git a/lib/Socket.h b/lib/Socket.h new file mode 100644 index 0000000..84def40 --- /dev/null +++ b/lib/Socket.h @@ -0,0 +1,322 @@ +/* + Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009 Her Majesty the + Queen in Right of Canada (Communications Research Center Canada) + + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. +*/ + +#pragma once + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif + +#include "ThreadsafeQueue.h" +#include <cstdlib> +#include <iostream> +#include <vector> +#include <atomic> +#include <thread> +#include <list> + +#include <sys/socket.h> +#include <netinet/in.h> +#include <unistd.h> +#include <netdb.h> +#include <arpa/inet.h> +#include <pthread.h> +#define SOCKET int +#define INVALID_SOCKET -1 +#define SOCKET_ERROR -1 + + +namespace Socket { + +struct InetAddress { + struct sockaddr_storage addr = {}; + + struct sockaddr *as_sockaddr() { return reinterpret_cast<sockaddr*>(&addr); }; + + void resolveUdpDestination(const std::string& destination, int port); +}; + +/** This class represents a UDP packet. + * + * A UDP packet contains a payload (sequence of bytes) and an address. For + * outgoing packets, the address is the destination address. For incoming + * packets, the address tells the user from what source the packet arrived from. + */ +class UDPPacket +{ + public: + UDPPacket(); + UDPPacket(size_t initSize); + + std::vector<uint8_t> buffer; + InetAddress address; +}; + +/** + * This class represents a socket for sending and receiving UDP packets. + * + * A UDP socket is the sending or receiving point for a packet delivery service. + * Each packet sent or received on a datagram socket is individually + * addressed and routed. Multiple packets sent from one machine to another may + * be routed differently, and may arrive in any order. + */ +class UDPSocket +{ + public: + /** Create a new socket that will not be bound to any port. To be used + * for data output. + */ + UDPSocket(); + /** Create a new socket. + * @param port The port number on which the socket will be bound + */ + UDPSocket(int port); + /** Create a new socket. + * @param port The port number on which the socket will be bound + * @param name The IP address on which the socket will be bound. + * It is used to bind the socket on a specific interface if + * the computer have many NICs. + */ + UDPSocket(int port, const std::string& name); + ~UDPSocket(); + UDPSocket(const UDPSocket& other) = delete; + const UDPSocket& operator=(const UDPSocket& other) = delete; + + /** Close the already open socket, and create a new one. Throws a runtime_error on error. */ + void reinit(int port); + void reinit(int port, const std::string& name); + + void close(void); + void send(UDPPacket& packet); + void send(const std::vector<uint8_t>& data, InetAddress destination); + UDPPacket receive(size_t max_size); + void joinGroup(const char* groupname, const char* if_addr = nullptr); + void setMulticastSource(const char* source_addr); + void setMulticastTTL(int ttl); + + /** Set blocking mode. By default, the socket is blocking. + * throws a runtime_error on error. + */ + void setBlocking(bool block); + + protected: + SOCKET m_sock; +}; + +/* Threaded UDP receiver */ +class UDPReceiver { + public: + UDPReceiver(); + ~UDPReceiver(); + UDPReceiver(const UDPReceiver&) = delete; + UDPReceiver operator=(const UDPReceiver&) = delete; + + // Start the receiver in a separate thread + void start(int port, const std::string& bindto, const std::string& mcastaddr, size_t max_packets_queued); + + // Get the data contained in a UDP packet, blocks if none available + // In case of error, throws a runtime_error + std::vector<uint8_t> get_packet_buffer(void); + + private: + void m_run(void); + + int m_port = 0; + std::string m_bindto; + std::string m_mcastaddr; + size_t m_max_packets_queued = 1; + std::thread m_thread; + std::atomic<bool> m_stop = ATOMIC_VAR_INIT(false); + ThreadsafeQueue<UDPPacket> m_packets; + UDPSocket m_sock; +}; + +class TCPSocket { + public: + TCPSocket(); + ~TCPSocket(); + TCPSocket(const TCPSocket& other) = delete; + TCPSocket& operator=(const TCPSocket& other) = delete; + TCPSocket(TCPSocket&& other); + TCPSocket& operator=(TCPSocket&& other); + + bool valid(void) const; + void connect(const std::string& hostname, int port, bool nonblock = false); + void listen(int port, const std::string& name); + void close(void); + + /* throws a runtime_error on failure, an invalid socket on timeout */ + TCPSocket accept(int timeout_ms); + + /* returns -1 on error, doesn't work on nonblocking sockets */ + ssize_t sendall(const void *buffer, size_t buflen); + + /** Send data over the TCP connection. + * @param data The buffer that will be sent. + * @param size Number of bytes to send. + * @param timeout_ms number of milliseconds before timeout, or 0 for infinite timeout + * return number of bytes sent, 0 on timeout, or throws runtime_error. + */ + ssize_t send(const void* data, size_t size, int timeout_ms=0); + + /* Returns number of bytes read, 0 on disconnect. Throws a + * runtime_error on error */ + ssize_t recv(void *buffer, size_t length, int flags); + + class Timeout {}; + class Interrupted {}; + /* Returns number of bytes read, 0 on disconnect or refused connection. + * Throws a Timeout on timeout, Interrupted on EINTR, a runtime_error + * on error + */ + ssize_t recv(void *buffer, size_t length, int flags, int timeout_ms); + + private: + explicit TCPSocket(int sockfd); + explicit TCPSocket(int sockfd, InetAddress remote_address); + SOCKET m_sock = -1; + + InetAddress m_remote_address; + + friend class TCPClient; +}; + +/* Implements a TCP receiver that auto-reconnects on errors */ +class TCPClient { + public: + void connect(const std::string& hostname, int port); + + /* Returns numer of bytes read, 0 on auto-reconnect, -1 + * on interruption. + * Throws a runtime_error on error */ + ssize_t recv(void *buffer, size_t length, int flags, int timeout_ms); + + private: + void reconnect(void); + TCPSocket m_sock; + std::string m_hostname; + int m_port; +}; + +/* Helper class for TCPDataDispatcher, contains a queue of pending data and + * a sender thread. */ +class TCPConnection +{ + public: + TCPConnection(TCPSocket&& sock); + TCPConnection(const TCPConnection&) = delete; + TCPConnection& operator=(const TCPConnection&) = delete; + ~TCPConnection(); + + ThreadsafeQueue<std::vector<uint8_t> > queue; + + private: + std::atomic<bool> m_running; + std::thread m_sender_thread; + TCPSocket m_sock; + + void process(void); +}; + +/* Send a TCP stream to several destinations, and automatically disconnect destinations + * whose buffer overflows. + */ +class TCPDataDispatcher +{ + public: + TCPDataDispatcher(size_t max_queue_size); + ~TCPDataDispatcher(); + TCPDataDispatcher(const TCPDataDispatcher&) = delete; + TCPDataDispatcher& operator=(const TCPDataDispatcher&) = delete; + + void start(int port, const std::string& address); + void write(const std::vector<uint8_t>& data); + + private: + void process(); + + size_t m_max_queue_size; + + std::atomic<bool> m_running = ATOMIC_VAR_INIT(false); + std::string m_exception_data; + std::thread m_listener_thread; + TCPSocket m_listener_socket; + std::list<TCPConnection> m_connections; +}; + +/* A TCP Server to receive data, which abstracts the handling of connects and disconnects. + */ +class TCPReceiveServer { + public: + TCPReceiveServer(size_t blocksize); + ~TCPReceiveServer(); + TCPReceiveServer(const TCPReceiveServer&) = delete; + TCPReceiveServer& operator=(const TCPReceiveServer&) = delete; + + void start(int listen_port, const std::string& address); + + // Return a vector that contains up to blocksize bytes of data, or + // and empty vector if no data is available. + std::vector<uint8_t> receive(); + + private: + void process(); + + size_t m_blocksize = 0; + ThreadsafeQueue<std::vector<uint8_t> > m_queue; + std::atomic<bool> m_running = ATOMIC_VAR_INIT(false); + std::string m_exception_data; + std::thread m_listener_thread; + TCPSocket m_listener_socket; +}; + +/* A TCP client that abstracts the handling of connects and disconnects. + */ +class TCPSendClient { + public: + TCPSendClient(const std::string& hostname, int port); + ~TCPSendClient(); + + /* Throws a runtime_error on error + */ + void sendall(const std::vector<uint8_t>& buffer); + + private: + void process(); + + std::string m_hostname; + int m_port; + + bool m_is_connected = false; + + TCPSocket m_sock; + static constexpr size_t MAX_QUEUE_SIZE = 1024; + ThreadsafeQueue<std::vector<uint8_t> > m_queue; + std::atomic<bool> m_running; + std::string m_exception_data; + std::thread m_sender_thread; + TCPSocket m_listener_socket; +}; + +} diff --git a/lib/ThreadsafeQueue.h b/lib/ThreadsafeQueue.h new file mode 100644 index 0000000..815dfe0 --- /dev/null +++ b/lib/ThreadsafeQueue.h @@ -0,0 +1,191 @@ +/* + Copyright (C) 2007, 2008, 2009, 2010, 2011 Her Majesty the Queen in + Right of Canada (Communications Research Center Canada) + + Copyright (C) 2018 + Matthias P. Braendli, matthias.braendli@mpb.li + + An implementation for a threadsafe queue, depends on C++11 + + When creating a ThreadsafeQueue, one can specify the minimal number + of elements it must contain before it is possible to take one + element out. + */ +/* + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + */ + +#pragma once + +#include <mutex> +#include <condition_variable> +#include <queue> +#include <utility> + +/* This queue is meant to be used by two threads. One producer + * that pushes elements into the queue, and one consumer that + * retrieves the elements. + * + * The queue can make the consumer block until an element + * is available, or a wakeup requested. + */ + +/* Class thrown by blocking pop to tell the consumer + * that there's a wakeup requested. */ +class ThreadsafeQueueWakeup {}; + +template<typename T> +class ThreadsafeQueue +{ +public: + /* Push one element into the queue, and notify another thread that + * might be waiting. + * + * if max_size > 0 and the queue already contains at least max_size elements, + * the element gets discarded. + * + * returns the new queue size. + */ + size_t push(T const& val, size_t max_size = 0) + { + std::unique_lock<std::mutex> lock(the_mutex); + size_t queue_size_before = the_queue.size(); + if (max_size == 0) { + the_queue.push(val); + } + else if (queue_size_before < max_size) { + the_queue.push(val); + } + size_t queue_size = the_queue.size(); + lock.unlock(); + + the_rx_notification.notify_one(); + + return queue_size; + } + + size_t push(T&& val, size_t max_size = 0) + { + std::unique_lock<std::mutex> lock(the_mutex); + size_t queue_size_before = the_queue.size(); + if (max_size == 0) { + the_queue.emplace(std::move(val)); + } + else if (queue_size_before < max_size) { + the_queue.emplace(std::move(val)); + } + size_t queue_size = the_queue.size(); + lock.unlock(); + + the_rx_notification.notify_one(); + + return queue_size; + } + + /* Push one element into the queue, but wait until the + * queue size goes below the threshold. + * + * Notify waiting thread. + * + * returns the new queue size. + */ + size_t push_wait_if_full(T const& val, size_t threshold) + { + std::unique_lock<std::mutex> lock(the_mutex); + while (the_queue.size() >= threshold) { + the_tx_notification.wait(lock); + } + the_queue.push(val); + size_t queue_size = the_queue.size(); + lock.unlock(); + + the_rx_notification.notify_one(); + + return queue_size; + } + + /* Trigger a wakeup event on a blocking consumer, which + * will receive a ThreadsafeQueueWakeup exception. + */ + void trigger_wakeup(void) + { + std::unique_lock<std::mutex> lock(the_mutex); + wakeup_requested = true; + lock.unlock(); + the_rx_notification.notify_one(); + } + + /* Send a notification for the receiver thread */ + void notify(void) + { + the_rx_notification.notify_one(); + } + + bool empty() const + { + std::unique_lock<std::mutex> lock(the_mutex); + return the_queue.empty(); + } + + size_t size() const + { + std::unique_lock<std::mutex> lock(the_mutex); + return the_queue.size(); + } + + bool try_pop(T& popped_value) + { + std::unique_lock<std::mutex> lock(the_mutex); + if (the_queue.empty()) { + return false; + } + + popped_value = the_queue.front(); + the_queue.pop(); + + lock.unlock(); + the_tx_notification.notify_one(); + + return true; + } + + void wait_and_pop(T& popped_value, size_t prebuffering = 1) + { + std::unique_lock<std::mutex> lock(the_mutex); + while (the_queue.size() < prebuffering and + not wakeup_requested) { + the_rx_notification.wait(lock); + } + + if (wakeup_requested) { + wakeup_requested = false; + throw ThreadsafeQueueWakeup(); + } + else { + std::swap(popped_value, the_queue.front()); + the_queue.pop(); + + lock.unlock(); + the_tx_notification.notify_one(); + } + } + +private: + std::queue<T> the_queue; + mutable std::mutex the_mutex; + std::condition_variable the_rx_notification; + std::condition_variable the_tx_notification; + bool wakeup_requested = false; +}; + diff --git a/lib/crc.c b/lib/crc.c new file mode 100644 index 0000000..cc02473 --- /dev/null +++ b/lib/crc.c @@ -0,0 +1,266 @@ +/* + Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009 Her Majesty the + Queen in Right of Canada (Communications Research Center Canada) + */ +/* + This file is part of ODR-DabMux. + + ODR-DabMux is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + ODR-DabMux is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with ODR-DabMux. If not, see <http://www.gnu.org/licenses/>. + */ + +#include "crc.h" +#ifndef _WIN32 +# include <unistd.h> +# include <netinet/in.h> +#endif +#include <stdio.h> +#include <fcntl.h> + +//#define CCITT 0x1021 + +uint8_t crc8tab[256] = { + 0x00, 0x07, 0x0e, 0x09, 0x1c, 0x1b, 0x12, 0x15, + 0x38, 0x3f, 0x36, 0x31, 0x24, 0x23, 0x2a, 0x2d, + 0x70, 0x77, 0x7e, 0x79, 0x6c, 0x6b, 0x62, 0x65, + 0x48, 0x4f, 0x46, 0x41, 0x54, 0x53, 0x5a, 0x5d, + 0xe0, 0xe7, 0xee, 0xe9, 0xfc, 0xfb, 0xf2, 0xf5, + 0xd8, 0xdf, 0xd6, 0xd1, 0xc4, 0xc3, 0xca, 0xcd, + 0x90, 0x97, 0x9e, 0x99, 0x8c, 0x8b, 0x82, 0x85, + 0xa8, 0xaf, 0xa6, 0xa1, 0xb4, 0xb3, 0xba, 0xbd, + 0xc7, 0xc0, 0xc9, 0xce, 0xdb, 0xdc, 0xd5, 0xd2, + 0xff, 0xf8, 0xf1, 0xf6, 0xe3, 0xe4, 0xed, 0xea, + 0xb7, 0xb0, 0xb9, 0xbe, 0xab, 0xac, 0xa5, 0xa2, + 0x8f, 0x88, 0x81, 0x86, 0x93, 0x94, 0x9d, 0x9a, + 0x27, 0x20, 0x29, 0x2e, 0x3b, 0x3c, 0x35, 0x32, + 0x1f, 0x18, 0x11, 0x16, 0x03, 0x04, 0x0d, 0x0a, + 0x57, 0x50, 0x59, 0x5e, 0x4b, 0x4c, 0x45, 0x42, + 0x6f, 0x68, 0x61, 0x66, 0x73, 0x74, 0x7d, 0x7a, + 0x89, 0x8e, 0x87, 0x80, 0x95, 0x92, 0x9b, 0x9c, + 0xb1, 0xb6, 0xbf, 0xb8, 0xad, 0xaa, 0xa3, 0xa4, + 0xf9, 0xfe, 0xf7, 0xf0, 0xe5, 0xe2, 0xeb, 0xec, + 0xc1, 0xc6, 0xcf, 0xc8, 0xdd, 0xda, 0xd3, 0xd4, + 0x69, 0x6e, 0x67, 0x60, 0x75, 0x72, 0x7b, 0x7c, + 0x51, 0x56, 0x5f, 0x58, 0x4d, 0x4a, 0x43, 0x44, + 0x19, 0x1e, 0x17, 0x10, 0x05, 0x02, 0x0b, 0x0c, + 0x21, 0x26, 0x2f, 0x28, 0x3d, 0x3a, 0x33, 0x34, + 0x4e, 0x49, 0x40, 0x47, 0x52, 0x55, 0x5c, 0x5b, + 0x76, 0x71, 0x78, 0x7f, 0x6a, 0x6d, 0x64, 0x63, + 0x3e, 0x39, 0x30, 0x37, 0x22, 0x25, 0x2c, 0x2b, + 0x06, 0x01, 0x08, 0x0f, 0x1a, 0x1d, 0x14, 0x13, + 0xae, 0xa9, 0xa0, 0xa7, 0xb2, 0xb5, 0xbc, 0xbb, + 0x96, 0x91, 0x98, 0x9f, 0x8a, 0x8d, 0x84, 0x83, + 0xde, 0xd9, 0xd0, 0xd7, 0xc2, 0xc5, 0xcc, 0xcb, + 0xe6, 0xe1, 0xe8, 0xef, 0xfa, 0xfd, 0xf4, 0xf3 +}; + + +uint16_t crc16tab[256] = { + 0x0000, 0x1021, 0x2042, 0x3063, 0x4084, 0x50a5, 0x60c6, 0x70e7, + 0x8108, 0x9129, 0xa14a, 0xb16b, 0xc18c, 0xd1ad, 0xe1ce, 0xf1ef, + 0x1231, 0x0210, 0x3273, 0x2252, 0x52b5, 0x4294, 0x72f7, 0x62d6, + 0x9339, 0x8318, 0xb37b, 0xa35a, 0xd3bd, 0xc39c, 0xf3ff, 0xe3de, + 0x2462, 0x3443, 0x0420, 0x1401, 0x64e6, 0x74c7, 0x44a4, 0x5485, + 0xa56a, 0xb54b, 0x8528, 0x9509, 0xe5ee, 0xf5cf, 0xc5ac, 0xd58d, + 0x3653, 0x2672, 0x1611, 0x0630, 0x76d7, 0x66f6, 0x5695, 0x46b4, + 0xb75b, 0xa77a, 0x9719, 0x8738, 0xf7df, 0xe7fe, 0xd79d, 0xc7bc, + 0x48c4, 0x58e5, 0x6886, 0x78a7, 0x0840, 0x1861, 0x2802, 0x3823, + 0xc9cc, 0xd9ed, 0xe98e, 0xf9af, 0x8948, 0x9969, 0xa90a, 0xb92b, + 0x5af5, 0x4ad4, 0x7ab7, 0x6a96, 0x1a71, 0x0a50, 0x3a33, 0x2a12, + 0xdbfd, 0xcbdc, 0xfbbf, 0xeb9e, 0x9b79, 0x8b58, 0xbb3b, 0xab1a, + 0x6ca6, 0x7c87, 0x4ce4, 0x5cc5, 0x2c22, 0x3c03, 0x0c60, 0x1c41, + 0xedae, 0xfd8f, 0xcdec, 0xddcd, 0xad2a, 0xbd0b, 0x8d68, 0x9d49, + 0x7e97, 0x6eb6, 0x5ed5, 0x4ef4, 0x3e13, 0x2e32, 0x1e51, 0x0e70, + 0xff9f, 0xefbe, 0xdfdd, 0xcffc, 0xbf1b, 0xaf3a, 0x9f59, 0x8f78, + 0x9188, 0x81a9, 0xb1ca, 0xa1eb, 0xd10c, 0xc12d, 0xf14e, 0xe16f, + 0x1080, 0x00a1, 0x30c2, 0x20e3, 0x5004, 0x4025, 0x7046, 0x6067, + 0x83b9, 0x9398, 0xa3fb, 0xb3da, 0xc33d, 0xd31c, 0xe37f, 0xf35e, + 0x02b1, 0x1290, 0x22f3, 0x32d2, 0x4235, 0x5214, 0x6277, 0x7256, + 0xb5ea, 0xa5cb, 0x95a8, 0x8589, 0xf56e, 0xe54f, 0xd52c, 0xc50d, + 0x34e2, 0x24c3, 0x14a0, 0x0481, 0x7466, 0x6447, 0x5424, 0x4405, + 0xa7db, 0xb7fa, 0x8799, 0x97b8, 0xe75f, 0xf77e, 0xc71d, 0xd73c, + 0x26d3, 0x36f2, 0x0691, 0x16b0, 0x6657, 0x7676, 0x4615, 0x5634, + 0xd94c, 0xc96d, 0xf90e, 0xe92f, 0x99c8, 0x89e9, 0xb98a, 0xa9ab, + 0x5844, 0x4865, 0x7806, 0x6827, 0x18c0, 0x08e1, 0x3882, 0x28a3, + 0xcb7d, 0xdb5c, 0xeb3f, 0xfb1e, 0x8bf9, 0x9bd8, 0xabbb, 0xbb9a, + 0x4a75, 0x5a54, 0x6a37, 0x7a16, 0x0af1, 0x1ad0, 0x2ab3, 0x3a92, + 0xfd2e, 0xed0f, 0xdd6c, 0xcd4d, 0xbdaa, 0xad8b, 0x9de8, 0x8dc9, + 0x7c26, 0x6c07, 0x5c64, 0x4c45, 0x3ca2, 0x2c83, 0x1ce0, 0x0cc1, + 0xef1f, 0xff3e, 0xcf5d, 0xdf7c, 0xaf9b, 0xbfba, 0x8fd9, 0x9ff8, + 0x6e17, 0x7e36, 0x4e55, 0x5e74, 0x2e93, 0x3eb2, 0x0ed1, 0x1ef0 +}; + + +uint32_t crc32tab[256] = { + 0x00000000, 0x04c11db7, 0x09823b6e, 0x0d4326d9, + 0x130476dc, 0x17c56b6b, 0x1a864db2, 0x1e475005, + 0x2608edb8, 0x22c9f00f, 0x2f8ad6d6, 0x2b4bcb61, + 0x350c9b64, 0x31cd86d3, 0x3c8ea00a, 0x384fbdbd, + 0x4c11db70, 0x48d0c6c7, 0x4593e01e, 0x4152fda9, + 0x5f15adac, 0x5bd4b01b, 0x569796c2, 0x52568b75, + 0x6a1936c8, 0x6ed82b7f, 0x639b0da6, 0x675a1011, + 0x791d4014, 0x7ddc5da3, 0x709f7b7a, 0x745e66cd, + 0x9823b6e0, 0x9ce2ab57, 0x91a18d8e, 0x95609039, + 0x8b27c03c, 0x8fe6dd8b, 0x82a5fb52, 0x8664e6e5, + 0xbe2b5b58, 0xbaea46ef, 0xb7a96036, 0xb3687d81, + 0xad2f2d84, 0xa9ee3033, 0xa4ad16ea, 0xa06c0b5d, + 0xd4326d90, 0xd0f37027, 0xddb056fe, 0xd9714b49, + 0xc7361b4c, 0xc3f706fb, 0xceb42022, 0xca753d95, + 0xf23a8028, 0xf6fb9d9f, 0xfbb8bb46, 0xff79a6f1, + 0xe13ef6f4, 0xe5ffeb43, 0xe8bccd9a, 0xec7dd02d, + 0x34867077, 0x30476dc0, 0x3d044b19, 0x39c556ae, + 0x278206ab, 0x23431b1c, 0x2e003dc5, 0x2ac12072, + 0x128e9dcf, 0x164f8078, 0x1b0ca6a1, 0x1fcdbb16, + 0x018aeb13, 0x054bf6a4, 0x0808d07d, 0x0cc9cdca, + 0x7897ab07, 0x7c56b6b0, 0x71159069, 0x75d48dde, + 0x6b93dddb, 0x6f52c06c, 0x6211e6b5, 0x66d0fb02, + 0x5e9f46bf, 0x5a5e5b08, 0x571d7dd1, 0x53dc6066, + 0x4d9b3063, 0x495a2dd4, 0x44190b0d, 0x40d816ba, + 0xaca5c697, 0xa864db20, 0xa527fdf9, 0xa1e6e04e, + 0xbfa1b04b, 0xbb60adfc, 0xb6238b25, 0xb2e29692, + 0x8aad2b2f, 0x8e6c3698, 0x832f1041, 0x87ee0df6, + 0x99a95df3, 0x9d684044, 0x902b669d, 0x94ea7b2a, + 0xe0b41de7, 0xe4750050, 0xe9362689, 0xedf73b3e, + 0xf3b06b3b, 0xf771768c, 0xfa325055, 0xfef34de2, + 0xc6bcf05f, 0xc27dede8, 0xcf3ecb31, 0xcbffd686, + 0xd5b88683, 0xd1799b34, 0xdc3abded, 0xd8fba05a, + 0x690ce0ee, 0x6dcdfd59, 0x608edb80, 0x644fc637, + 0x7a089632, 0x7ec98b85, 0x738aad5c, 0x774bb0eb, + 0x4f040d56, 0x4bc510e1, 0x46863638, 0x42472b8f, + 0x5c007b8a, 0x58c1663d, 0x558240e4, 0x51435d53, + 0x251d3b9e, 0x21dc2629, 0x2c9f00f0, 0x285e1d47, + 0x36194d42, 0x32d850f5, 0x3f9b762c, 0x3b5a6b9b, + 0x0315d626, 0x07d4cb91, 0x0a97ed48, 0x0e56f0ff, + 0x1011a0fa, 0x14d0bd4d, 0x19939b94, 0x1d528623, + 0xf12f560e, 0xf5ee4bb9, 0xf8ad6d60, 0xfc6c70d7, + 0xe22b20d2, 0xe6ea3d65, 0xeba91bbc, 0xef68060b, + 0xd727bbb6, 0xd3e6a601, 0xdea580d8, 0xda649d6f, + 0xc423cd6a, 0xc0e2d0dd, 0xcda1f604, 0xc960ebb3, + 0xbd3e8d7e, 0xb9ff90c9, 0xb4bcb610, 0xb07daba7, + 0xae3afba2, 0xaafbe615, 0xa7b8c0cc, 0xa379dd7b, + 0x9b3660c6, 0x9ff77d71, 0x92b45ba8, 0x9675461f, + 0x8832161a, 0x8cf30bad, 0x81b02d74, 0x857130c3, + 0x5d8a9099, 0x594b8d2e, 0x5408abf7, 0x50c9b640, + 0x4e8ee645, 0x4a4ffbf2, 0x470cdd2b, 0x43cdc09c, + 0x7b827d21, 0x7f436096, 0x7200464f, 0x76c15bf8, + 0x68860bfd, 0x6c47164a, 0x61043093, 0x65c52d24, + 0x119b4be9, 0x155a565e, 0x18197087, 0x1cd86d30, + 0x029f3d35, 0x065e2082, 0x0b1d065b, 0x0fdc1bec, + 0x3793a651, 0x3352bbe6, 0x3e119d3f, 0x3ad08088, + 0x2497d08d, 0x2056cd3a, 0x2d15ebe3, 0x29d4f654, + 0xc5a92679, 0xc1683bce, 0xcc2b1d17, 0xc8ea00a0, + 0xd6ad50a5, 0xd26c4d12, 0xdf2f6bcb, 0xdbee767c, + 0xe3a1cbc1, 0xe760d676, 0xea23f0af, 0xeee2ed18, + 0xf0a5bd1d, 0xf464a0aa, 0xf9278673, 0xfde69bc4, + 0x89b8fd09, 0x8d79e0be, 0x803ac667, 0x84fbdbd0, + 0x9abc8bd5, 0x9e7d9662, 0x933eb0bb, 0x97ffad0c, + 0xafb010b1, 0xab710d06, 0xa6322bdf, 0xa2f33668, + 0xbcb4666d, 0xb8757bda, 0xb5365d03, 0xb1f740b4 +}; + +// This function can be used to create a new table with a different polynom +void init_crc8tab(uint8_t l_code, uint8_t l_init) +{ + unsigned i, j, msb; + uint8_t nb; + uint8_t crc; + + for (i = 0; i < 256; ++i) { + crc = l_init; + nb = i ^ 0xff; + for (j = 0; j < 8; ++j) { + msb = (nb & (0x80 >> j)) && 1; + msb ^= (crc >> 7); + crc <<= 1; + if (msb) + crc ^= l_code; + } + crc8tab[i] = crc; + } +} + + +void init_crc16tab(uint16_t l_code, uint16_t l_init) +{ + unsigned i, j, msb; + uint8_t nb; + uint16_t crc; + + for (i = 0; i < 256; ++i) { + crc = l_init; + nb = i ^ 0xff; + for (j = 0; j < 8; ++j) { + msb = (nb & (0x80 >> j)) && 1; + msb ^= (crc >> 15); + crc <<= 1; + if (msb) + crc ^= l_code; + } + crc ^= 0xff00; + crc16tab[i] = crc; + } +} + + +void init_crc32tab(uint32_t l_code, uint32_t l_init) +{ + unsigned i, j, msb; + uint8_t nb; + uint32_t crc; + + for (i = 0; i < 256; ++i) { + crc = l_init; + nb = i ^ 0xff; + for (j = 0; j < 8; ++j) { + msb = (nb & (0x80 >> j)) && 1; + msb ^= (crc >> 31); + crc <<= 1; + if (msb) + crc ^= l_code; + } + crc ^= 0xffffff00; + crc32tab[i] = crc; + } +} + + +uint8_t crc8(uint8_t l_crc, const void *lp_data, unsigned l_nb) +{ + const uint8_t* data = (const uint8_t*)lp_data; + while (l_nb--) { + l_crc = crc8tab[l_crc ^ *(data++)]; + } + return (l_crc); +} + + +uint16_t crc16(uint16_t l_crc, const void *lp_data, unsigned l_nb) +{ + const uint8_t* data = (const uint8_t*)lp_data; + while (l_nb--) { + l_crc = + (l_crc << 8) ^ crc16tab[(l_crc >> 8) ^ *(data++)]; + } + return (l_crc); +} + + +uint32_t crc32(uint32_t l_crc, const void *lp_data, unsigned l_nb) +{ + const uint8_t* data = (const uint8_t*)lp_data; + while (l_nb--) { + l_crc = + (l_crc << 8) ^ crc32tab[((l_crc >> 24) ^ *(data++)) & 0xff]; + } + return (l_crc); +} diff --git a/lib/crc.h b/lib/crc.h new file mode 100644 index 0000000..b1785a1 --- /dev/null +++ b/lib/crc.h @@ -0,0 +1,59 @@ +/* + Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009 Her Majesty the + Queen in Right of Canada (Communications Research Center Canada) + */ +/* + This file is part of ODR-DabMux. + + ODR-DabMux is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + ODR-DabMux is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with ODR-DabMux. If not, see <http://www.gnu.org/licenses/>. + */ + +#ifndef _CRC +#define _CRC + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif + +#ifndef _WIN32 + #include <stdint.h> +#else + #include <winsock2.h> // For types... + typedef BYTE uint8_t; + typedef WORD uint16_t; + typedef DWORD32 uint32_t; +#endif + + +#ifdef __cplusplus +extern "C" { // } +#endif + +void init_crc8tab(uint8_t l_code, uint8_t l_init); +uint8_t crc8(uint8_t l_crc, const void *lp_data, unsigned l_nb); +extern uint8_t crc8tab[]; + +void init_crc16tab(uint16_t l_code, uint16_t l_init); +uint16_t crc16(uint16_t l_crc, const void *lp_data, unsigned l_nb); +extern uint16_t crc16tab[]; + +void init_crc32tab(uint32_t l_code, uint32_t l_init); +uint32_t crc32(uint32_t l_crc, const void *lp_data, unsigned l_nb); +extern uint32_t crc32tab[]; + +#ifdef __cplusplus +} +#endif + +#endif //_CRC diff --git a/lib/edi/PFT.cpp b/lib/edi/PFT.cpp new file mode 100644 index 0000000..158b206 --- /dev/null +++ b/lib/edi/PFT.cpp @@ -0,0 +1,574 @@ +/* ------------------------------------------------------------------ + * Copyright (C) 2017 AVT GmbH - Fabien Vercasson + * Copyright (C) 2017 Matthias P. Braendli + * matthias.braendli@mpb.li + * + * http://opendigitalradio.org + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. + * See the License for the specific language governing permissions + * and limitations under the License. + * ------------------------------------------------------------------- + */ + +#include <cstdio> +#include <cassert> +#include <cstring> +#include <sstream> +#include <stdexcept> +#include <algorithm> +#include "crc.h" +#include "PFT.hpp" +#include "Log.h" +#include "buffer_unpack.hpp" +extern "C" { +#include "fec/fec.h" +} + +namespace EdiDecoder { +namespace PFT { + +using namespace std; + +const findex_t NUM_AFBUILDERS_TO_KEEP = 10; + +static bool checkCRC(const uint8_t *buf, size_t size) +{ + const uint16_t crc_from_packet = read_16b(buf + size - 2); + uint16_t crc_calc = 0xffff; + crc_calc = crc16(crc_calc, buf, size - 2); + crc_calc ^= 0xffff; + + return crc_from_packet == crc_calc; +} + +class FECDecoder { + public: + FECDecoder() { + m_rs_handler = init_rs_char( + symsize, gfPoly, firstRoot, primElem, nroots, pad); + } + FECDecoder(const FECDecoder& other) = delete; + FECDecoder& operator=(const FECDecoder& other) = delete; + ~FECDecoder() { + free_rs_char(m_rs_handler); + } + + // return -1 in case of failure, non-negative value if errors + // were corrected. + // Known positions of erasures should be given in eras_pos to + // improve decoding probability. After calling this function + // eras_pos will contain the positions of the corrected errors. + int decode(vector<uint8_t> &data, vector<int> &eras_pos) { + assert(data.size() == N); + const size_t no_eras = eras_pos.size(); + + eras_pos.resize(nroots); + int num_err = decode_rs_char(m_rs_handler, data.data(), + eras_pos.data(), no_eras); + if (num_err > 0) { + eras_pos.resize(num_err); + } + return num_err; + } + + // return -1 in case of failure, non-negative value if errors + // were corrected. No known erasures. + int decode(vector<uint8_t> &data) { + assert(data.size() == N); + int num_err = decode_rs_char(m_rs_handler, data.data(), nullptr, 0); + return num_err; + } + + private: + void* m_rs_handler; + + const int firstRoot = 1; // Discovered by analysing EDI dump + const int gfPoly = 0x11d; + + // The encoding has to be 255, 207 always, because the chunk has to + // be padded at the end, and not at the beginning as libfec would + // do + const size_t N = 255; + const size_t K = 207; + const int primElem = 1; + const int symsize = 8; + const size_t nroots = N - K; // For EDI PFT, this must be 48 + const size_t pad = ((1 << symsize) - 1) - N; // is 255-N + +}; + +size_t Fragment::loadData(const std::vector<uint8_t> &buf) +{ + const size_t header_len = 14; + if (buf.size() < header_len) { + return 0; + } + + size_t index = 0; + + // Parse PFT Fragment Header (ETSI TS 102 821 V1.4.1 ch7.1) + if (not (buf[0] == 'P' and buf[1] == 'F') ) { + throw invalid_argument("Invalid PFT SYNC bytes"); + } + index += 2; // Psync + + _Pseq = read_16b(buf.begin()+index); index += 2; + _Findex = read_24b(buf.begin()+index); index += 3; + _Fcount = read_24b(buf.begin()+index); index += 3; + _FEC = unpack1bit(buf[index], 0); + _Addr = unpack1bit(buf[index], 1); + _Plen = read_16b(buf.begin()+index) & 0x3FFF; index += 2; + + const size_t required_len = header_len + + (_FEC ? 1 : 0) + + (_Addr ? 2 : 0) + + 2; // CRC + if (buf.size() < required_len) { + return 0; + } + + // Optional RS Header + _RSk = 0; + _RSz = 0; + if (_FEC) { + _RSk = buf[index]; index += 1; + _RSz = buf[index]; index += 1; + } + + // Optional transport header + _Source = 0; + _Dest = 0; + if (_Addr) { + _Source = read_16b(buf.begin()+index); index += 2; + _Dest = read_16b(buf.begin()+index); index += 2; + } + + index += 2; + const bool crc_valid = checkCRC(buf.data(), index); + const bool buf_has_enough_data = (buf.size() >= index + _Plen); + + if (not buf_has_enough_data) { + return 0; + } + + _valid = ((not _FEC) or crc_valid) and buf_has_enough_data; + +#if 0 + if (!_valid) { + stringstream ss; + ss << "Invalid PF fragment: "; + if (_FEC) { + ss << " RSk=" << (uint32_t)_RSk << " RSz=" << (uint32_t)_RSz; + } + + if (_Addr) { + ss << " Source=" << _Source << " Dest=" << _Dest; + } + etiLog.log(debug, "%s\n", ss.str().c_str()); + } +#endif + + _payload.clear(); + if (_valid) { + copy( buf.begin()+index, + buf.begin()+index+_Plen, + back_inserter(_payload)); + index += _Plen; + } + + return index; +} + + +AFBuilder::AFBuilder(pseq_t Pseq, findex_t Fcount, size_t lifetime) +{ + _Pseq = Pseq; + _Fcount = Fcount; + assert(lifetime > 0); + lifeTime = lifetime; +} + +void AFBuilder::pushPFTFrag(const Fragment &frag) +{ + if (_Pseq != frag.Pseq() or _Fcount != frag.Fcount()) { + throw invalid_argument("Invalid PFT fragment Pseq or Fcount"); + } + const auto Findex = frag.Findex(); + const bool fragment_already_received = _fragments.count(Findex); + + if (not fragment_already_received) + { + _fragments[Findex] = frag; + } +} + +bool Fragment::checkConsistency(const Fragment& other) const +{ + /* Consistency check, TS 102 821 Clause 7.3.2. + * + * Every PFT Fragment produced from a single AF or RS Packet shall have + * the same values in all of the PFT Header fields except for the Findex, + * Plen and HCRC fields. + */ + + return other._Fcount == _Fcount and + other._FEC == _FEC and + other._RSk == _RSk and + other._RSz == _RSz and + other._Addr == _Addr and + other._Source == _Source and + other._Dest == _Dest and + + /* The Plen field of all fragments shall be the s for the initial f-1 + * fragments and s - (L%f) for the final fragment. + * Note that when Reed Solomon has been used, all fragments will be of + * length s. + */ + (_FEC ? other._Plen == _Plen : true); +} + + +AFBuilder::decode_attempt_result_t AFBuilder::canAttemptToDecode() const +{ + if (_fragments.empty()) { + return AFBuilder::decode_attempt_result_t::no; + } + + if (_fragments.size() == _Fcount) { + return AFBuilder::decode_attempt_result_t::yes; + } + + /* Check that all fragments are consistent */ + const Fragment& first = _fragments.begin()->second; + if (not std::all_of(_fragments.begin(), _fragments.end(), + [&](const pair<int, Fragment>& pair) { + const Fragment& frag = pair.second; + return first.checkConsistency(frag) and _Pseq == frag.Pseq(); + }) ) { + throw invalid_argument("Inconsistent PFT fragments"); + } + + // Calculate the minimum number of fragments necessary to apply FEC. + // This can't be done with the last fragment that may have a + // smaller size + // ETSI TS 102 821 V1.4.1 ch 7.4.4 + auto frag_it = _fragments.begin(); + if (frag_it->second.Fcount() == _Fcount - 1) { + frag_it++; + + if (frag_it == _fragments.end()) { + return AFBuilder::decode_attempt_result_t::no; + } + } + + const Fragment& frag = frag_it->second; + + if ( frag.FEC() ) + { + const uint16_t _Plen = frag.Plen(); + + /* max number of RS chunks that may have been sent */ + const uint32_t _cmax = (_Fcount*_Plen) / (frag.RSk()+48); + assert(_cmax > 0); + + /* Receiving _rxmin fragments does not guarantee that decoding + * will succeed! */ + const uint32_t _rxmin = _Fcount - (_cmax*48)/_Plen; + + if (_fragments.size() >= _rxmin) { + return AFBuilder::decode_attempt_result_t::maybe; + } + } + + return AFBuilder::decode_attempt_result_t::no; +} + +std::vector<uint8_t> AFBuilder::extractAF() const +{ + if (not _af_packet.empty()) { + return _af_packet; + } + + bool ok = false; + + if (canAttemptToDecode() != AFBuilder::decode_attempt_result_t::no) { + + auto frag_it = _fragments.begin(); + if (frag_it->second.Fcount() == _Fcount - 1) { + frag_it++; + + if (frag_it == _fragments.end()) { + throw std::runtime_error("Invalid attempt at extracting AF"); + } + } + + const Fragment& ref_frag = frag_it->second; + const auto RSk = ref_frag.RSk(); + const auto RSz = ref_frag.RSz(); + const auto Plen = ref_frag.Plen(); + + if ( ref_frag.FEC() ) + { + const uint32_t cmax = (_Fcount*Plen) / (RSk+48); + + // Keep track of erasures (missing fragments) for + // every chunk + map<int, vector<int> > erasures; + + + // Assemble fragments into a RS block, immediately + // deinterleaving it. + vector<uint8_t> rs_block(Plen * _Fcount); + for (size_t j = 0; j < _Fcount; j++) { + const bool fragment_present = _fragments.count(j); + if (fragment_present) { + const auto& fragment = _fragments.at(j).payload(); + + if (j != _Fcount - 1 and fragment.size() != Plen) { + throw runtime_error("Incorrect fragment length " + + to_string(fragment.size()) + " " + + to_string(Plen)); + } + + if (j == _Fcount - 1 and fragment.size() > Plen) { + throw runtime_error("Incorrect last fragment length " + + to_string(fragment.size()) + " " + + to_string(Plen)); + } + + size_t k = 0; + for (; k < fragment.size(); k++) { + rs_block[k * _Fcount + j] = fragment[k]; + } + + for (; k < Plen; k++) { + rs_block[k * _Fcount + j] = 0x00; + } + } + else { + // fill with zeros if fragment is missing + for (size_t k = 0; k < Plen; k++) { + rs_block[k * _Fcount + j] = 0x00; + + const size_t chunk_ix = (k * _Fcount + j) / (RSk + 48); + const size_t chunk_offset = (k * _Fcount + j) % (RSk + 48); + erasures[chunk_ix].push_back(chunk_offset); + } + } + } + + // The RS block is a concatenation of chunks of RSk bytes + 48 parity + // followed by RSz padding + + FECDecoder fec; + for (size_t i = 0; i < cmax; i++) { + // We need to pad the chunk ourself + vector<uint8_t> chunk(255); + const auto& block_begin = rs_block.begin() + (RSk + 48) * i; + copy(block_begin, block_begin + RSk, chunk.begin()); + // bytes between RSk and 207 are 0x00 already + copy(block_begin + RSk, block_begin + RSk + 48, + chunk.begin() + 207); + + int errors_corrected = -1; + if (erasures.count(i)) { + errors_corrected = fec.decode(chunk, erasures[i]); + } + else { + errors_corrected = fec.decode(chunk); + } + + if (errors_corrected == -1) { + _af_packet.clear(); + return {}; + } + +#if 0 + if (errors_corrected > 0) { + etiLog.log(debug, "Corrected %d errors at ", errors_corrected); + for (const auto &index : erasures[i]) { + etiLog.log(debug, " %d", index); + } + etiLog.log(debug, "\n"); + } +#endif + + _af_packet.insert(_af_packet.end(), chunk.begin(), chunk.begin() + RSk); + } + + _af_packet.resize(_af_packet.size() - RSz); + } + else { + // No FEC: just assemble fragments + + for (size_t j = 0; j < _Fcount; ++j) { + const bool fragment_present = _fragments.count(j); + if (fragment_present) + { + const auto& fragment = _fragments.at(j); + + _af_packet.insert(_af_packet.end(), + fragment.payload().begin(), + fragment.payload().end()); + } + else { + throw logic_error("Missing fragment"); + } + } + } + + // EDI specific, must have a CRC. + if( _af_packet.size() >= 12 ) { + ok = checkCRC(_af_packet.data(), _af_packet.size()); + + if (not ok) { + etiLog.log(debug, "Too many errors to reconstruct AF from %zu/%u" + " PFT fragments\n", _fragments.size(), _Fcount); + } + } + } + + if (not ok) { + _af_packet.clear(); + } + + return _af_packet; +} + +std::string AFBuilder::visualise() const +{ + stringstream ss; + ss << "|"; + for (size_t i = 0; i < _Fcount; i++) { + if (_fragments.count(i)) { + ss << "."; + } + else { + ss << " "; + } + } + ss << "| " << AFBuilder::dar_to_string(canAttemptToDecode()) << " " << lifeTime; + return ss.str(); +} + +void PFT::pushPFTFrag(const Fragment &fragment) +{ + // Start decoding the first pseq we receive. In normal + // operation without interruptions, the map should + // never become empty + if (m_afbuilders.empty()) { + m_next_pseq = fragment.Pseq(); + etiLog.log(debug,"Initialise next_pseq to %u\n", m_next_pseq); + } + + if (m_afbuilders.count(fragment.Pseq()) == 0) { + // The AFBuilder wants to know the lifetime in number of fragments, + // we know the delay in number of AF packets. Every AF packet + // is cut into Fcount fragments. + const size_t lifetime = fragment.Fcount() * m_max_delay; + + // Build the afbuilder in the map in-place + m_afbuilders.emplace(std::piecewise_construct, + /* key */ + std::forward_as_tuple(fragment.Pseq()), + /* builder */ + std::forward_as_tuple(fragment.Pseq(), fragment.Fcount(), lifetime)); + } + + auto& p = m_afbuilders.at(fragment.Pseq()); + p.pushPFTFrag(fragment); + + if (m_verbose) { + etiLog.log(debug, "Got frag %u:%u, afbuilders: ", + fragment.Pseq(), fragment.Findex()); + for (const auto &k : m_afbuilders) { + const bool isNextPseq = (m_next_pseq == k.first); + etiLog.level(debug) << (isNextPseq ? "->" : " ") << + k.first << " " << k.second.visualise(); + } + } +} + + +std::vector<uint8_t> PFT::getNextAFPacket() +{ + if (m_afbuilders.count(m_next_pseq) == 0) { + if (m_afbuilders.size() > m_max_delay) { + m_afbuilders.clear(); + etiLog.level(debug) << " Reinit"; + } + + return {}; + } + + auto &builder = m_afbuilders.at(m_next_pseq); + + using dar_t = AFBuilder::decode_attempt_result_t; + + if (builder.canAttemptToDecode() == dar_t::yes) { + auto afpacket = builder.extractAF(); + assert(not afpacket.empty()); + incrementNextPseq(); + return afpacket; + } + else if (builder.canAttemptToDecode() == dar_t::maybe) { + if (builder.lifeTime > 0) { + builder.lifeTime--; + } + + if (builder.lifeTime == 0) { + // Attempt Reed-Solomon decoding + auto afpacket = builder.extractAF(); + + if (afpacket.empty()) { + etiLog.log(debug,"pseq %d timed out after RS", m_next_pseq); + } + incrementNextPseq(); + return afpacket; + } + } + else { + if (builder.lifeTime > 0) { + builder.lifeTime--; + } + + if (builder.lifeTime == 0) { + etiLog.log(debug, "pseq %d timed out\n", m_next_pseq); + incrementNextPseq(); + } + } + + return {}; +} + +void PFT::setMaxDelay(size_t num_af_packets) +{ + m_max_delay = num_af_packets; +} + +void PFT::setVerbose(bool enable) +{ + m_verbose = enable; +} + +void PFT::incrementNextPseq() +{ + if (m_afbuilders.count(m_next_pseq - NUM_AFBUILDERS_TO_KEEP) > 0) { + m_afbuilders.erase(m_next_pseq - NUM_AFBUILDERS_TO_KEEP); + } + + m_next_pseq++; +} + +} +} diff --git a/lib/edi/PFT.hpp b/lib/edi/PFT.hpp new file mode 100644 index 0000000..208fd70 --- /dev/null +++ b/lib/edi/PFT.hpp @@ -0,0 +1,167 @@ +/* ------------------------------------------------------------------ + * Copyright (C) 2017 AVT GmbH - Fabien Vercasson + * Copyright (C) 2017 Matthias P. Braendli + * matthias.braendli@mpb.li + * + * http://opendigitalradio.org + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. + * See the License for the specific language governing permissions + * and limitations under the License. + * ------------------------------------------------------------------- + */ + +#pragma once +#include <cstdio> +#include <cstdint> +#include <vector> +#include <map> +#include <string> + +namespace EdiDecoder { +namespace PFT { + +using pseq_t = uint16_t; +using findex_t = uint32_t; // findex is a 24-bit value + +class Fragment +{ + public: + // Load the data for one fragment from buf into + // the Fragment. + // \returns the number of bytes of useful data found in buf + // A non-zero return value doesn't imply a valid fragment + // the isValid() method must be used to verify this. + size_t loadData(const std::vector<uint8_t> &buf); + + bool isValid() const { return _valid; } + pseq_t Pseq() const { return _Pseq; } + findex_t Findex() const { return _Findex; } + findex_t Fcount() const { return _Fcount; } + bool FEC() const { return _FEC; } + uint16_t Plen() const { return _Plen; } + uint8_t RSk() const { return _RSk; } + uint8_t RSz() const { return _RSz; } + const std::vector<uint8_t>& payload() const + { return _payload; } + + bool checkConsistency(const Fragment& other) const; + + private: + std::vector<uint8_t> _payload; + + pseq_t _Pseq = 0; + findex_t _Findex = 0; + findex_t _Fcount = 0; + bool _FEC = false; + bool _Addr = false; + uint16_t _Plen = 0; + uint8_t _RSk = 0; + uint8_t _RSz = 0; + uint16_t _Source = 0; + uint16_t _Dest = 0; + bool _valid = false; +}; + +/* The AFBuilder collects Fragments and builds an Application Frame + * out of them. It does error correction if necessary + */ +class AFBuilder +{ + public: + enum class decode_attempt_result_t { + yes, // The AF packet can be build because all fragments are present + maybe, // RS decoding may correctly decode the AF packet + no, // Not enough fragments present to permit RS + }; + + static std::string dar_to_string(decode_attempt_result_t dar) { + switch (dar) { + case decode_attempt_result_t::yes: return "y"; + case decode_attempt_result_t::no: return "n"; + case decode_attempt_result_t::maybe: return "m"; + } + return "?"; + } + + AFBuilder(pseq_t Pseq, findex_t Fcount, size_t lifetime); + + void pushPFTFrag(const Fragment &frag); + + /* Assess if it may be possible to decode this AF packet */ + decode_attempt_result_t canAttemptToDecode() const; + + /* Try to build the AF with received fragments. + * Apply error correction if necessary (missing packets/CRC errors) + * \return an empty vector if building the AF is not possible + */ + std::vector<uint8_t> extractAF(void) const; + + std::pair<findex_t, findex_t> + numberOfFragments(void) const { + return {_fragments.size(), _Fcount}; + } + + std::string visualise(void) const; + + /* The user of this instance can keep track of the lifetime of this + * builder + */ + size_t lifeTime; + + private: + + // A map from fragment index to fragment + std::map<findex_t, Fragment> _fragments; + + // cached version of decoded AF packet + mutable std::vector<uint8_t> _af_packet; + + pseq_t _Pseq; + findex_t _Fcount; +}; + +class PFT +{ + public: + void pushPFTFrag(const Fragment &fragment); + + /* Try to build the AF packet for the next pseq. This might + * skip one or more pseq according to the maximum delay setting. + * + * \return an empty vector if building the AF is not possible + */ + std::vector<uint8_t> getNextAFPacket(void); + + /* Set the maximum delay in number of AF Packets before we + * abandon decoding a given pseq. + */ + void setMaxDelay(size_t num_af_packets); + + /* Enable verbose fprintf */ + void setVerbose(bool enable); + + private: + void incrementNextPseq(void); + + pseq_t m_next_pseq; + size_t m_max_delay = 10; // in AF packets + + // Keep one AFBuilder for each Pseq + std::map<pseq_t, AFBuilder> m_afbuilders; + + bool m_verbose = 0; +}; + +} + +} diff --git a/lib/edi/STIDecoder.cpp b/lib/edi/STIDecoder.cpp new file mode 100644 index 0000000..d55cc12 --- /dev/null +++ b/lib/edi/STIDecoder.cpp @@ -0,0 +1,226 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://opendigitalradio.org + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ +#include "STIDecoder.hpp" +#include "buffer_unpack.hpp" +#include "crc.h" +#include "Log.h" +#include <cstdio> +#include <cassert> +#include <sstream> + +namespace EdiDecoder { + +using namespace std; + +STIDecoder::STIDecoder(STIDataCollector& data_collector, bool verbose) : + m_data_collector(data_collector), + m_dispatcher(std::bind(&STIDecoder::packet_completed, this), verbose) +{ + using std::placeholders::_1; + using std::placeholders::_2; + m_dispatcher.register_tag("*ptr", + std::bind(&STIDecoder::decode_starptr, this, _1, _2)); + m_dispatcher.register_tag("dsti", + std::bind(&STIDecoder::decode_dsti, this, _1, _2)); + m_dispatcher.register_tag("ss", + std::bind(&STIDecoder::decode_ssn, this, _1, _2)); + m_dispatcher.register_tag("*dmy", + std::bind(&STIDecoder::decode_stardmy, this, _1, _2)); + m_dispatcher.register_tag("ODRa", + std::bind(&STIDecoder::decode_odraudiolevel, this, _1, _2)); + m_dispatcher.register_tag("ODRv", + std::bind(&STIDecoder::decode_odrversion, this, _1, _2)); +} + +void STIDecoder::push_bytes(const vector<uint8_t> &buf) +{ + m_dispatcher.push_bytes(buf); +} + +void STIDecoder::push_packet(const vector<uint8_t> &buf) +{ + m_dispatcher.push_packet(buf); +} + +void STIDecoder::setMaxDelay(int num_af_packets) +{ + m_dispatcher.setMaxDelay(num_af_packets); +} + +#define AFPACKET_HEADER_LEN 10 // includes SYNC + +bool STIDecoder::decode_starptr(const vector<uint8_t> &value, uint16_t) +{ + if (value.size() != 0x40 / 8) { + etiLog.log(warn, "Incorrect length %02lx for *PTR", value.size()); + return false; + } + + char protocol_sz[5]; + protocol_sz[4] = '\0'; + copy(value.begin(), value.begin() + 4, protocol_sz); + string protocol(protocol_sz); + + uint16_t major = read_16b(value.begin() + 4); + uint16_t minor = read_16b(value.begin() + 6); + + m_data_collector.update_protocol(protocol, major, minor); + + return true; +} + +bool STIDecoder::decode_dsti(const vector<uint8_t> &value, uint16_t) +{ + size_t offset = 0; + + const uint16_t dstiHeader = read_16b(value.begin() + offset); + offset += 2; + + sti_management_data md; + + md.stihf = (dstiHeader >> 15) & 0x1; + md.atstf = (dstiHeader >> 14) & 0x1; + md.rfadf = (dstiHeader >> 13) & 0x1; + uint8_t dfcth = (dstiHeader >> 8) & 0x1F; + uint8_t dfctl = dstiHeader & 0xFF; + + md.dflc = dfcth * 250 + dfctl; // modulo 5000 counter + + const size_t expected_length = 2 + + (md.stihf ? 3 : 0) + + (md.atstf ? 1 + 4 + 3 : 0) + + (md.rfadf ? 9 : 0); + + if (value.size() != expected_length) { + throw std::runtime_error("EDI dsti: decoding error:" + "value.size() != expected_length: " + + to_string(value.size()) + " " + + to_string(expected_length)); + } + + if (md.stihf) { + const uint8_t stat = value[offset++]; + const uint16_t spid = read_16b(value.begin() + offset); + m_data_collector.update_stat(stat, spid); + offset += 2; + } + + if (md.atstf) { + uint8_t utco = value[offset]; + offset++; + + uint32_t seconds = read_32b(value.begin() + offset); + offset += 4; + + m_data_collector.update_edi_time(utco, seconds); + + md.tsta = read_24b(value.begin() + offset); + offset += 3; + } + else { + // Null timestamp, ETSI ETS 300 799, C.2.2 + md.tsta = 0xFFFFFF; + } + + + if (md.rfadf) { + std::array<uint8_t, 9> rfad; + copy(value.cbegin() + offset, + value.cbegin() + offset + 9, + rfad.begin()); + offset += 9; + + m_data_collector.update_rfad(rfad); + } + + m_data_collector.update_sti_management(md); + + return true; +} + +bool STIDecoder::decode_ssn(const vector<uint8_t> &value, uint16_t n) +{ + sti_payload_data sti; + + sti.stream_index = n - 1; // n is 1-indexed + sti.rfa = value[0] >> 3; + sti.tid = value[0] & 0x07; + + uint16_t istc = read_24b(value.begin() + 1); + sti.tidext = istc >> 13; + sti.crcstf = (istc >> 12) & 0x1; + sti.stid = istc & 0xFFF; + + if (sti.rfa != 0) { + etiLog.level(warn) << "EDI: rfa field in SSnn tag non-null"; + } + + copy( value.cbegin() + 3, + value.cend(), + back_inserter(sti.istd)); + + m_data_collector.add_payload(move(sti)); + + return true; +} + +bool STIDecoder::decode_stardmy(const vector<uint8_t>& /*value*/, uint16_t) +{ + return true; +} + +bool STIDecoder::decode_odraudiolevel(const vector<uint8_t>& value, uint16_t) +{ + constexpr size_t expected_length = 2 * sizeof(int16_t); + + audio_level_data audio_level; + + if (value.size() == expected_length) { + audio_level.left = read_16b(value.begin()); + audio_level.right = read_16b(value.begin() + 2); + } + else { + audio_level.left = 0; + audio_level.right = 0; + etiLog.level(warn) << "EDI: ODR AudioLevel TAG has wrong length!"; + } + + m_data_collector.update_audio_levels(audio_level); + + // Not being able to decode the audio level is a soft-failure, it should + // not disrupt decoding the actual audio data. + return true; +} + +bool STIDecoder::decode_odrversion(const vector<uint8_t>& value, uint16_t) +{ + const auto vd = parse_odr_version_data(value); + m_data_collector.update_odr_version(vd); + + return true; +} + +void STIDecoder::packet_completed() +{ + m_data_collector.assemble(); +} + +} diff --git a/lib/edi/STIDecoder.hpp b/lib/edi/STIDecoder.hpp new file mode 100644 index 0000000..e2aa850 --- /dev/null +++ b/lib/edi/STIDecoder.hpp @@ -0,0 +1,134 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://opendigitalradio.org + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ +#pragma once + +#include "common.hpp" +#include <cstdint> +#include <deque> +#include <string> +#include <vector> +#include <array> + +namespace EdiDecoder { + +// Information for STI-D Management +struct sti_management_data { + bool stihf; + bool atstf; + bool rfadf; + uint16_t dflc; + + uint32_t tsta; +}; + +// Information for a subchannel available in EDI +struct sti_payload_data { + uint16_t stream_index; + uint8_t rfa; + uint8_t tid; + uint8_t tidext; + bool crcstf; + uint16_t stid; + std::vector<uint8_t> istd; + + // Return the length of ISTD in bytes + uint16_t stl(void) const { return istd.size(); } +}; + +struct audio_level_data { + int16_t left = 0; + int16_t right = 0; +}; + + +/* A class that receives STI data must implement the interface described + * in the STIDataCollector. This can be e.g. a converter to ETI, or something that + * prepares data structures for a modulator. + */ +class STIDataCollector { + public: + // Tell the ETIWriter what EDI protocol we receive in *ptr. + // This is not part of the ETI data, but is used as check + virtual void update_protocol( + const std::string& proto, + uint16_t major, + uint16_t minor) = 0; + + // STAT error field and service provider ID + virtual void update_stat(uint8_t stat, uint16_t spid) = 0; + + // In addition to TSTA in ETI, EDI also transports more time + // stamp information. + virtual void update_edi_time(uint32_t utco, uint32_t seconds) = 0; + + virtual void update_rfad(std::array<uint8_t, 9> rfad) = 0; + virtual void update_sti_management(const sti_management_data& data) = 0; + + virtual void add_payload(sti_payload_data&& payload) = 0; + + virtual void update_audio_levels(const audio_level_data& data) = 0; + virtual void update_odr_version(const odr_version_data& data) = 0; + + virtual void assemble() = 0; +}; + +/* The STIDecoder takes care of decoding the EDI TAGs related to the transport + * of ETI(NI) data inside AF and PF packets. + * + * PF packets are handed over to the PFT decoder, which will in turn return + * AF packets. AF packets are directly handled (TAG extraction) here. + */ +class STIDecoder { + public: + STIDecoder(STIDataCollector& data_collector, bool verbose); + + /* Push bytes into the decoder. The buf can contain more + * than a single packet. This is useful when reading from streams + * (files, TCP) + */ + void push_bytes(const std::vector<uint8_t> &buf); + + /* Push a complete packet into the decoder. Useful for UDP and other + * datagram-oriented protocols. + */ + void push_packet(const std::vector<uint8_t> &buf); + + /* Set the maximum delay in number of AF Packets before we + * abandon decoding a given pseq. + */ + void setMaxDelay(int num_af_packets); + + private: + bool decode_starptr(const std::vector<uint8_t> &value, uint16_t); + bool decode_dsti(const std::vector<uint8_t> &value, uint16_t); + bool decode_ssn(const std::vector<uint8_t> &value, uint16_t n); + bool decode_stardmy(const std::vector<uint8_t> &value, uint16_t); + + bool decode_odraudiolevel(const std::vector<uint8_t> &value, uint16_t); + bool decode_odrversion(const std::vector<uint8_t> &value, uint16_t); + + void packet_completed(); + + STIDataCollector& m_data_collector; + TagDispatcher m_dispatcher; +}; + +} diff --git a/lib/edi/STIWriter.cpp b/lib/edi/STIWriter.cpp new file mode 100644 index 0000000..a7e4f20 --- /dev/null +++ b/lib/edi/STIWriter.cpp @@ -0,0 +1,142 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://opendigitalradio.org + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ +#include "STIWriter.hpp" +#include "crc.h" +#include "Log.h" +#include <cstdio> +#include <cassert> +#include <stdexcept> +#include <sstream> +#include <ctime> +#include <iostream> +#include <iomanip> +#include <sstream> + +namespace EdiDecoder { + +using namespace std; + +STIWriter::STIWriter(std::function<void(sti_frame_t&&)>&& frame_callback) : + m_frame_callback(move(frame_callback)) +{ +} + +void STIWriter::update_protocol( + const std::string& proto, + uint16_t major, + uint16_t minor) +{ + m_proto_valid = (proto == "DSTI" and major == 0 and minor == 0); + + if (not m_proto_valid) { + throw std::invalid_argument("Wrong EDI protocol"); + } +} + + +void STIWriter::update_stat(uint8_t stat, uint16_t spid) +{ + m_stat = stat; + m_spid = spid; + m_stat_valid = true; + + if (m_stat != 0xFF) { + etiLog.log(warn, "STI errorlevel %02x", m_stat); + } +} + +void STIWriter::update_rfad(std::array<uint8_t, 9> rfad) +{ + (void)rfad; +} + +void STIWriter::update_sti_management(const sti_management_data& data) +{ + m_management_data = data; + m_management_data_valid = true; +} + +void STIWriter::add_payload(sti_payload_data&& payload) +{ + m_payload = move(payload); + m_payload_valid = true; +} + +void STIWriter::update_audio_levels(const audio_level_data& data) +{ + m_audio_levels = data; +} + +void STIWriter::update_odr_version(const odr_version_data& data) +{ + m_version_data = data; +} + +void STIWriter::update_edi_time( + uint32_t utco, + uint32_t seconds) +{ + if (not m_proto_valid) { + throw std::runtime_error("Cannot update time before protocol"); + } + + m_utco = utco; + m_seconds = seconds; + + // TODO check validity + m_time_valid = true; +} + + +void STIWriter::assemble() +{ + if (not m_proto_valid) { + throw std::runtime_error("Cannot assemble STI before protocol"); + } + + if (not m_management_data_valid) { + throw std::runtime_error("Cannot assemble STI before management data"); + } + + if (not m_payload_valid) { + throw std::runtime_error("Cannot assemble STI without frame data"); + } + + // TODO check time validity + + sti_frame_t stiFrame; + stiFrame.frame = move(m_payload.istd); + stiFrame.timestamp.seconds = m_seconds; + stiFrame.timestamp.utco = m_utco; + stiFrame.timestamp.tsta = m_management_data.tsta; + stiFrame.audio_levels = m_audio_levels; + stiFrame.version_data = m_version_data; + + m_frame_callback(move(stiFrame)); + + m_proto_valid = false; + m_management_data_valid = false; + m_stat_valid = false; + m_time_valid = false; + m_payload_valid = false; +} + +} diff --git a/lib/edi/STIWriter.hpp b/lib/edi/STIWriter.hpp new file mode 100644 index 0000000..fc08e97 --- /dev/null +++ b/lib/edi/STIWriter.hpp @@ -0,0 +1,90 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://opendigitalradio.org + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ +#pragma once + +#include "common.hpp" +#include "STIDecoder.hpp" +#include <cstdint> +#include <functional> +#include <string> +#include <vector> +#include <list> + +namespace EdiDecoder { + +struct sti_frame_t { + std::vector<uint8_t> frame; + frame_timestamp_t timestamp; + audio_level_data audio_levels; + odr_version_data version_data; +}; + +class STIWriter : public STIDataCollector { + public: + // The callback gets called for every STI frame that gets assembled + STIWriter(std::function<void(sti_frame_t&&)>&& frame_callback); + + // Tell the ETIWriter what EDI protocol we receive in *ptr. + // This is not part of the ETI data, but is used as check + virtual void update_protocol( + const std::string& proto, + uint16_t major, + uint16_t minor); + + virtual void update_stat(uint8_t stat, uint16_t spid); + + virtual void update_edi_time( + uint32_t utco, + uint32_t seconds); + + virtual void update_rfad(std::array<uint8_t, 9> rfad); + virtual void update_sti_management(const sti_management_data& data); + virtual void add_payload(sti_payload_data&& payload); + + virtual void update_audio_levels(const audio_level_data& data); + virtual void update_odr_version(const odr_version_data& data); + + virtual void assemble(void); + private: + std::function<void(sti_frame_t&&)> m_frame_callback; + + bool m_proto_valid = false; + + bool m_management_data_valid = false; + sti_management_data m_management_data; + + bool m_stat_valid = false; + uint8_t m_stat = 0; + uint16_t m_spid = 0; + + bool m_time_valid = false; + uint32_t m_utco = 0; + uint32_t m_seconds = 0; + + bool m_payload_valid = false; + sti_payload_data m_payload; + + audio_level_data m_audio_levels; + odr_version_data m_version_data; +}; + +} + diff --git a/lib/edi/buffer_unpack.hpp b/lib/edi/buffer_unpack.hpp new file mode 100644 index 0000000..a996017 --- /dev/null +++ b/lib/edi/buffer_unpack.hpp @@ -0,0 +1,62 @@ +/* + Copyright (C) 2016 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://opendigitalradio.org + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#pragma once +#include <cstdint> + +namespace EdiDecoder { + +template<class T> +uint16_t read_16b(T buf) +{ + uint16_t value = 0; + value = (uint16_t)(buf[0]) << 8; + value |= (uint16_t)(buf[1]); + return value; +} + +template<class T> +uint32_t read_24b(T buf) +{ + uint32_t value = 0; + value = (uint32_t)(buf[0]) << 16; + value |= (uint32_t)(buf[1]) << 8; + value |= (uint32_t)(buf[2]); + return value; +} + +template<class T> +uint32_t read_32b(T buf) +{ + uint32_t value = 0; + value = (uint32_t)(buf[0]) << 24; + value |= (uint32_t)(buf[1]) << 16; + value |= (uint32_t)(buf[2]) << 8; + value |= (uint32_t)(buf[3]); + return value; +} + +inline uint32_t unpack1bit(uint8_t byte, int bitpos) +{ + return (byte & 1 << (7-bitpos)) > (7-bitpos); +} + +} diff --git a/lib/edi/common.cpp b/lib/edi/common.cpp new file mode 100644 index 0000000..470b3ba --- /dev/null +++ b/lib/edi/common.cpp @@ -0,0 +1,385 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://opendigitalradio.org + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ +#include "common.hpp" +#include "buffer_unpack.hpp" +#include "Log.h" +#include "crc.h" +#include <sstream> +#include <cassert> +#include <cmath> +#include <cstdio> + +namespace EdiDecoder { + +using namespace std; + +bool frame_timestamp_t::valid() const +{ + return tsta != 0xFFFFFF; +} + +string frame_timestamp_t::to_string() const +{ + const time_t seconds_in_unix_epoch = to_unix_epoch(); + + stringstream ss; + if (valid()) { + ss << "Timestamp: "; + } + else { + ss << "Timestamp not valid: "; + } + + char timestr[100]; + if (std::strftime(timestr, sizeof(timestr), "%Y-%m-%dZ%H:%M:%S", std::gmtime(&seconds_in_unix_epoch))) { + ss << timestr << " + " << ((double)tsta / 16384000.0); + } + else { + ss << "unknown"; + } + return ss.str(); +} + +time_t frame_timestamp_t::to_unix_epoch() const +{ + // EDI epoch: 2000-01-01T00:00:00Z + // Convert using + // TZ=UTC python -c 'import datetime; print(datetime.datetime(2000,1,1,0,0,0,0).strftime("%s"))' + return 946684800 + seconds - utco; +} + +double frame_timestamp_t::diff_ms(const frame_timestamp_t& other) const +{ + const double lhs = (double)seconds + (tsta / 16384000.0); + const double rhs = (double)other.seconds + (other.tsta / 16384000.0); + return lhs - rhs; +} + +frame_timestamp_t& frame_timestamp_t::operator+=(const std::chrono::milliseconds& ms) +{ + tsta += (ms.count() % 1000) << 14; // Shift ms by 14 to Timestamp level 2 + if (tsta > 0xf9FFff) { + tsta -= 0xfa0000; // Substract 16384000, corresponding to one second + seconds += 1; + } + + seconds += (ms.count() / 1000); + + return *this; +} + +frame_timestamp_t frame_timestamp_t::from_unix_epoch(std::time_t time, uint32_t tai_utc_offset, uint32_t tsta) +{ + frame_timestamp_t ts; + + const std::time_t posix_timestamp_1_jan_2000 = 946684800; + + ts.utco = tai_utc_offset - 32; + ts.seconds = time - posix_timestamp_1_jan_2000 + ts.utco; + ts.tsta = tsta; + return ts; +} + +std::chrono::system_clock::time_point frame_timestamp_t::to_system_clock() const +{ + auto ts = chrono::system_clock::from_time_t(to_unix_epoch()); + + // PPS offset in seconds = tsta / 16384000 + // We cannot use nanosecond resolution because not all platforms use a + // system_clock that has nanosecond precision. It's not really important, + // as this function is only used for debugging. + ts += chrono::microseconds(std::lrint(tsta / 16.384)); + + return ts; +} + + +TagDispatcher::TagDispatcher( + std::function<void()>&& af_packet_completed, bool verbose) : + m_af_packet_completed(move(af_packet_completed)) +{ + m_pft.setVerbose(verbose); +} + +void TagDispatcher::push_bytes(const vector<uint8_t> &buf) +{ + copy(buf.begin(), buf.end(), back_inserter(m_input_data)); + + while (m_input_data.size() > 2) { + if (m_input_data[0] == 'A' and m_input_data[1] == 'F') { + const decode_state_t st = decode_afpacket(m_input_data); + + if (st.num_bytes_consumed == 0 and not st.complete) { + // We need to refill our buffer + break; + } + + if (st.num_bytes_consumed) { + vector<uint8_t> remaining_data; + copy(m_input_data.begin() + st.num_bytes_consumed, + m_input_data.end(), + back_inserter(remaining_data)); + m_input_data = remaining_data; + } + + if (st.complete) { + m_af_packet_completed(); + } + } + else if (m_input_data[0] == 'P' and m_input_data[1] == 'F') { + PFT::Fragment fragment; + const size_t fragment_bytes = fragment.loadData(m_input_data); + + if (fragment_bytes == 0) { + // We need to refill our buffer + break; + } + + vector<uint8_t> remaining_data; + copy(m_input_data.begin() + fragment_bytes, + m_input_data.end(), + back_inserter(remaining_data)); + m_input_data = remaining_data; + + if (fragment.isValid()) { + m_pft.pushPFTFrag(fragment); + } + + auto af = m_pft.getNextAFPacket(); + if (not af.empty()) { + decode_state_t st = decode_afpacket(af); + + if (st.complete) { + m_af_packet_completed(); + } + } + } + else { + etiLog.log(warn,"Unknown %c!", *m_input_data.data()); + m_input_data.erase(m_input_data.begin()); + } + } +} + +void TagDispatcher::push_packet(const vector<uint8_t> &buf) +{ + if (buf.size() < 2) { + throw std::invalid_argument("Not enough bytes to read EDI packet header"); + } + + if (buf[0] == 'A' and buf[1] == 'F') { + const decode_state_t st = decode_afpacket(buf); + + if (st.complete) { + m_af_packet_completed(); + } + + } + else if (buf[0] == 'P' and buf[1] == 'F') { + PFT::Fragment fragment; + fragment.loadData(buf); + + if (fragment.isValid()) { + m_pft.pushPFTFrag(fragment); + } + + auto af = m_pft.getNextAFPacket(); + if (not af.empty()) { + const decode_state_t st = decode_afpacket(af); + + if (st.complete) { + m_af_packet_completed(); + } + } + } + else { + const char packettype[3] = {(char)buf[0], (char)buf[1], '\0'}; + std::stringstream ss; + ss << "Unknown EDI packet "; + ss << packettype; + throw std::invalid_argument(ss.str()); + } +} + +void TagDispatcher::setMaxDelay(int num_af_packets) +{ + m_pft.setMaxDelay(num_af_packets); +} + + +#define AFPACKET_HEADER_LEN 10 // includes SYNC +decode_state_t TagDispatcher::decode_afpacket( + const std::vector<uint8_t> &input_data) +{ + if (input_data.size() < AFPACKET_HEADER_LEN) { + return {false, 0}; + } + + // read length from packet + uint32_t taglength = read_32b(input_data.begin() + 2); + uint16_t seq = read_16b(input_data.begin() + 6); + + const size_t crclength = 2; + if (input_data.size() < AFPACKET_HEADER_LEN + taglength + crclength) { + return {false, 0}; + } + + // SEQ wraps at 0xFFFF, unsigned integer overflow is intentional + const uint16_t expected_seq = m_last_seq + 1; + if (expected_seq != seq) { + etiLog.level(warn) << "EDI AF Packet sequence error, " << seq; + } + m_last_seq = seq; + + bool has_crc = (input_data[8] & 0x80) ? true : false; + uint8_t major_revision = (input_data[8] & 0x70) >> 4; + uint8_t minor_revision = input_data[8] & 0x0F; + if (major_revision != 1 or minor_revision != 0) { + throw invalid_argument("EDI AF Packet has wrong revision " + + to_string(major_revision) + "." + to_string(minor_revision)); + } + uint8_t pt = input_data[9]; + if (pt != 'T') { + // only support Tag + return {false, 0}; + } + + + if (not has_crc) { + throw invalid_argument("AF packet not supported, has no CRC"); + } + + uint16_t crc = 0xffff; + for (size_t i = 0; i < AFPACKET_HEADER_LEN + taglength; i++) { + crc = crc16(crc, &input_data[i], 1); + } + crc ^= 0xffff; + + uint16_t packet_crc = read_16b(input_data.begin() + AFPACKET_HEADER_LEN + taglength); + + if (packet_crc != crc) { + throw invalid_argument( + "AF Packet crc wrong"); + } + else { + vector<uint8_t> payload(taglength); + copy(input_data.begin() + AFPACKET_HEADER_LEN, + input_data.begin() + AFPACKET_HEADER_LEN + taglength, + payload.begin()); + + return {decode_tagpacket(payload), + AFPACKET_HEADER_LEN + taglength + 2}; + } +} + +void TagDispatcher::register_tag(const std::string& tag, tag_handler&& h) +{ + m_handlers[tag] = move(h); +} + + +bool TagDispatcher::decode_tagpacket(const vector<uint8_t> &payload) +{ + size_t length = 0; + + bool success = true; + + for (size_t i = 0; i + 8 < payload.size(); i += 8 + length) { + char tag_sz[5]; + tag_sz[4] = '\0'; + copy(payload.begin() + i, payload.begin() + i + 4, tag_sz); + + string tag(tag_sz); + + uint32_t taglength = read_32b(payload.begin() + i + 4); + + if (taglength % 8 != 0) { + etiLog.log(warn, "Invalid EDI tag length, not multiple of 8!"); + break; + } + taglength /= 8; + + length = taglength; + + const size_t calculated_length = i + 8 + taglength; + if (calculated_length > payload.size()) { + etiLog.log(warn, "Invalid EDI tag length: tag larger %zu than tagpacket %zu!", + calculated_length, payload.size()); + break; + } + + vector<uint8_t> tag_value(taglength); + copy( payload.begin() + i+8, + payload.begin() + i+8+taglength, + tag_value.begin()); + + bool tagsuccess = false; + bool found = false; + for (auto tag_handler : m_handlers) { + if (tag_handler.first.size() == 4 and tag_handler.first == tag) { + found = true; + tagsuccess = tag_handler.second(tag_value, 0); + } + else if (tag_handler.first.size() == 3 and + tag.substr(0, 3) == tag_handler.first) { + found = true; + uint8_t n = tag_sz[3]; + tagsuccess = tag_handler.second(tag_value, n); + } + else if (tag_handler.first.size() == 2 and + tag.substr(0, 2) == tag_handler.first) { + found = true; + uint16_t n = 0; + n = (uint16_t)(tag_sz[2]) << 8; + n |= (uint16_t)(tag_sz[3]); + tagsuccess = tag_handler.second(tag_value, n); + } + } + + if (not found) { + etiLog.log(warn, "Ignoring unknown TAG %s", tag.c_str()); + break; + } + + if (not tagsuccess) { + etiLog.log(warn, "Error decoding TAG %s", tag.c_str()); + success = tagsuccess; + break; + } + } + + return success; +} + +odr_version_data parse_odr_version_data(const std::vector<uint8_t>& data) +{ + if (data.size() < sizeof(uint32_t)) { + return {}; + } + + const size_t versionstr_length = data.size() - sizeof(uint32_t); + string version(data.begin(), data.begin() + versionstr_length); + uint32_t uptime_s = read_32b(data.begin() + versionstr_length); + + return {version, uptime_s}; +} + +} diff --git a/lib/edi/common.hpp b/lib/edi/common.hpp new file mode 100644 index 0000000..2a9c683 --- /dev/null +++ b/lib/edi/common.hpp @@ -0,0 +1,106 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://opendigitalradio.org + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ +#pragma once + +#include "PFT.hpp" +#include <functional> +#include <map> +#include <chrono> +#include <string> +#include <vector> +#include <cstddef> +#include <ctime> + +namespace EdiDecoder { + +struct frame_timestamp_t { + uint32_t seconds = 0; + uint32_t utco = 0; + uint32_t tsta = 0; // According to EN 300 797 Annex B + + bool valid() const; + std::string to_string() const; + std::time_t to_unix_epoch() const; + std::chrono::system_clock::time_point to_system_clock() const; + + double diff_ms(const frame_timestamp_t& other) const; + + frame_timestamp_t& operator+=(const std::chrono::milliseconds& ms); + + static frame_timestamp_t from_unix_epoch(std::time_t time, uint32_t tai_utc_offset, uint32_t tsta); +}; + +struct decode_state_t { + decode_state_t(bool _complete, size_t _num_bytes_consumed) : + complete(_complete), num_bytes_consumed(_num_bytes_consumed) {} + bool complete; + size_t num_bytes_consumed; +}; + +/* The TagDispatcher takes care of decoding EDI, with or without PFT, and + * will call functions when TAGs are encountered. + * + * PF packets are handed over to the PFT decoder, which will in turn return + * AF packets. AF packets are directly dispatched to the TAG functions. + */ +class TagDispatcher { + public: + TagDispatcher(std::function<void()>&& af_packet_completed, bool verbose); + + /* Push bytes into the decoder. The buf can contain more + * than a single packet. This is useful when reading from streams + * (files, TCP) + */ + void push_bytes(const std::vector<uint8_t> &buf); + + /* Push a complete packet into the decoder. Useful for UDP and other + * datagram-oriented protocols. + */ + void push_packet(const std::vector<uint8_t> &buf); + + /* Set the maximum delay in number of AF Packets before we + * abandon decoding a given pseq. + */ + void setMaxDelay(int num_af_packets); + + using tag_handler = std::function<bool(std::vector<uint8_t>, uint16_t)>; + void register_tag(const std::string& tag, tag_handler&& h); + + private: + decode_state_t decode_afpacket(const std::vector<uint8_t> &input_data); + bool decode_tagpacket(const std::vector<uint8_t> &payload); + + PFT::PFT m_pft; + uint16_t m_last_seq = 0; + std::vector<uint8_t> m_input_data; + std::map<std::string, tag_handler> m_handlers; + std::function<void()> m_af_packet_completed; +}; + +// Data carried inside the ODRv EDI TAG +struct odr_version_data { + std::string version; + uint32_t uptime_s; +}; + +odr_version_data parse_odr_version_data(const std::vector<uint8_t>& data); + +} diff --git a/lib/edioutput/AFPacket.cpp b/lib/edioutput/AFPacket.cpp new file mode 100644 index 0000000..b38c38b --- /dev/null +++ b/lib/edioutput/AFPacket.cpp @@ -0,0 +1,96 @@ +/* + Copyright (C) 2014 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output. + This implements an AF Packet as defined ETSI TS 102 821. + Also see ETSI TS 102 693 + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ +#include "config.h" +#include "crc.h" +#include "AFPacket.h" +#include "TagItems.h" +#include "TagPacket.h" +#include <vector> +#include <string> +#include <iostream> +#include <cstdio> +#include <cstdint> +#include <arpa/inet.h> + +namespace edi { + +// Header PT field. AF packet contains TAG payload +const uint8_t AFHEADER_PT_TAG = 'T'; + +// AF Packet Major (3 bits) and Minor (4 bits) version +const uint8_t AFHEADER_VERSION = 0x10; // MAJ=1, MIN=0 + +AFPacket AFPacketiser::Assemble(TagPacket tag_packet) +{ + std::vector<uint8_t> payload = tag_packet.Assemble(); + + if (m_verbose) + std::cerr << "Assemble AFPacket " << seq << std::endl; + + std::string pack_data("AF"); // SYNC + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + + uint32_t taglength = payload.size(); + + if (m_verbose) + std::cerr << " AFPacket payload size " << payload.size() << std::endl; + + // write length into packet + packet.push_back((taglength >> 24) & 0xFF); + packet.push_back((taglength >> 16) & 0xFF); + packet.push_back((taglength >> 8) & 0xFF); + packet.push_back(taglength & 0xFF); + + // fill rest of header + packet.push_back(seq >> 8); + packet.push_back(seq & 0xFF); + seq++; + packet.push_back((have_crc ? 0x80 : 0) | AFHEADER_VERSION); // ar_cf: CRC=1 + packet.push_back(AFHEADER_PT_TAG); + + // insert payload, must have a length multiple of 8 bytes + packet.insert(packet.end(), payload.begin(), payload.end()); + + // calculate CRC over AF Header and payload + uint16_t crc = 0xffff; + crc = crc16(crc, &(packet.front()), packet.size()); + crc ^= 0xffff; + + if (m_verbose) + fprintf(stderr, " AFPacket crc %x\n", crc); + + packet.push_back((crc >> 8) & 0xFF); + packet.push_back(crc & 0xFF); + + if (m_verbose) + std::cerr << " AFPacket length " << packet.size() << std::endl; + + return packet; +} + +} diff --git a/lib/edioutput/AFPacket.h b/lib/edioutput/AFPacket.h new file mode 100644 index 0000000..f2c4e35 --- /dev/null +++ b/lib/edioutput/AFPacket.h @@ -0,0 +1,61 @@ +/* + Copyright (C) 2014 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output. + This implements an AF Packet as defined ETSI TS 102 821. + Also see ETSI TS 102 693 + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#include "config.h" +#include <vector> +#include <cstdint> +#include "TagItems.h" +#include "TagPacket.h" + +namespace edi { + +typedef std::vector<uint8_t> AFPacket; + +// ETSI TS 102 821, 6.1 AF packet structure +class AFPacketiser +{ + public: + AFPacketiser() : + m_verbose(false) {}; + AFPacketiser(bool verbose) : + m_verbose(verbose) {}; + + AFPacket Assemble(TagPacket tag_packet); + + private: + static const bool have_crc = true; + + uint16_t seq = 0; //counter that overflows at 0xFFFF + + bool m_verbose; +}; + +} + diff --git a/lib/edioutput/EDIConfig.h b/lib/edioutput/EDIConfig.h new file mode 100644 index 0000000..ca76322 --- /dev/null +++ b/lib/edioutput/EDIConfig.h @@ -0,0 +1,84 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output, + UDP and TCP transports and their configuration + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#include "config.h" +#include <vector> +#include <string> +#include <memory> +#include <cstdint> + +namespace edi { + +/** Configuration for EDI output */ + +struct destination_t { + virtual ~destination_t() {}; +}; + +// Can represent both unicast and multicast destinations +struct udp_destination_t : public destination_t { + std::string dest_addr; + std::string source_addr; + unsigned int source_port = 0; + unsigned int ttl = 10; +}; + +// TCP server that can accept multiple connections +struct tcp_server_t : public destination_t { + unsigned int listen_port = 0; + size_t max_frames_queued = 1024; +}; + +// TCP client that connects to one endpoint +struct tcp_client_t : public destination_t { + std::string dest_addr; + unsigned int dest_port = 0; + size_t max_frames_queued = 1024; +}; + +struct configuration_t { + unsigned chunk_len = 207; // RSk, data length of each chunk + unsigned fec = 0; // number of fragments that can be recovered + bool dump = false; // dump a file with the EDI packets + bool verbose = false; + bool enable_pft = false; // Enable protection and fragmentation + unsigned int tagpacket_alignment = 0; + std::vector<std::shared_ptr<destination_t> > destinations; + unsigned int dest_port = 0; // common destination port, because it's encoded in the transport layer + unsigned int latency_frames = 0; // if nonzero, enable interleaver with a latency of latency_frames * 24ms + + bool enabled() const { return destinations.size() > 0; } + bool interleaver_enabled() const { return latency_frames > 0; } + + void print() const; +}; + +} + + diff --git a/lib/edioutput/Interleaver.cpp b/lib/edioutput/Interleaver.cpp new file mode 100644 index 0000000..f26a50e --- /dev/null +++ b/lib/edioutput/Interleaver.cpp @@ -0,0 +1,122 @@ +/* + Copyright (C) 2017 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output, + Interleaving of PFT fragments to increase robustness against + burst packet loss. + + This is possible because EDI has to assume that fragments may reach + the receiver out of order. + + */ +/* + This file is part of ODR-DabMux. + + ODR-DabMux is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + ODR-DabMux is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with ODR-DabMux. If not, see <http://www.gnu.org/licenses/>. + */ + +#include "Interleaver.h" +#include <cassert> + +namespace edi { + +void Interleaver::SetLatency(size_t latency_frames) +{ + m_latency = latency_frames; +} + +Interleaver::fragment_vec Interleaver::Interleave(fragment_vec &fragments) +{ + m_fragment_count = fragments.size(); + + // Create vectors containing Fcount*latency fragments in total + // and store them into the deque + if (m_buffer.empty()) { + m_buffer.emplace_back(); + } + + auto& last_buffer = m_buffer.back(); + + for (auto& fragment : fragments) { + const bool last_buffer_is_complete = + (last_buffer.size() >= m_fragment_count * m_latency); + + if (last_buffer_is_complete) { + m_buffer.emplace_back(); + last_buffer = m_buffer.back(); + } + + last_buffer.push_back(std::move(fragment)); + } + + fragments.clear(); + + while ( not m_buffer.empty() and + (m_buffer.front().size() >= m_fragment_count * m_latency)) { + + auto& first_buffer = m_buffer.front(); + + assert(first_buffer.size() == m_fragment_count * m_latency); + + /* Assume we have 5 fragments per AF frame, and latency of 3. + * This will give the following strides: + * 0 1 2 + * +-------+-------+---+ + * | 0 1 | 2 3 | 4 | + * | | +---+ | + * | 5 6 | 7 | 8 9 | + * | +---+ | | + * |10 |11 12 |13 14 | + * +---+-------+-------+ + * + * ix will be 0, 5, 10, 1, 6 in the first loop + */ + + for (size_t i = 0; i < m_fragment_count; i++) { + const size_t ix = m_interleave_offset + m_fragment_count * m_stride; + m_interleaved_fragments.push_back(first_buffer.at(ix)); + + m_stride += 1; + if (m_stride >= m_latency) { + m_interleave_offset++; + m_stride = 0; + } + } + + if (m_interleave_offset >= m_fragment_count) { + m_interleave_offset = 0; + m_stride = 0; + m_buffer.pop_front(); + } + } + + std::vector<PFTFragment> interleaved_frags; + + const size_t n = std::min(m_fragment_count, m_interleaved_fragments.size()); + std::move(m_interleaved_fragments.begin(), + m_interleaved_fragments.begin() + n, + std::back_inserter(interleaved_frags)); + m_interleaved_fragments.erase( + m_interleaved_fragments.begin(), + m_interleaved_fragments.begin() + n); + + return interleaved_frags; +} + +} + + diff --git a/lib/edioutput/Interleaver.h b/lib/edioutput/Interleaver.h new file mode 100644 index 0000000..3029d5d --- /dev/null +++ b/lib/edioutput/Interleaver.h @@ -0,0 +1,75 @@ +/* + Copyright (C) 2017 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output, + Interleaving of PFT fragments to increase robustness against + burst packet loss. + + This is possible because EDI has to assume that fragments may reach + the receiver out of order. + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#include "config.h" +#include <vector> +#include <deque> +#include <stdexcept> +#include <cstdint> +#include "Log.h" +#include "PFT.h" + +namespace edi { + +class Interleaver { + public: + using fragment_vec = std::vector<PFTFragment>; + + /* Configure the interleaver to use latency_frames number of AF + * packets for interleaving. Total delay through the interleaver + * will be latency_frames * 24ms + */ + void SetLatency(size_t latency_frames); + + /* Move the fragments for an AF Packet into the interleaver and + * return interleaved fragments to be transmitted. + */ + fragment_vec Interleave(fragment_vec &fragments); + + private: + size_t m_latency = 0; + size_t m_fragment_count = 0; + size_t m_interleave_offset = 0; + size_t m_stride = 0; + + /* Buffer that accumulates enough fragments to interleave */ + std::deque<fragment_vec> m_buffer; + + /* Buffer that contains fragments that have been interleaved, + * to avoid that the interleaver output is too bursty + */ + std::deque<PFTFragment> m_interleaved_fragments; +}; + +} + diff --git a/lib/edioutput/PFT.cpp b/lib/edioutput/PFT.cpp new file mode 100644 index 0000000..371d36f --- /dev/null +++ b/lib/edioutput/PFT.cpp @@ -0,0 +1,327 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output, + Protection, Fragmentation and Transport. (PFT) + + Are supported: + Reed-Solomon and Fragmentation + + This implements part of PFT as defined ETSI TS 102 821. + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include "config.h" +#include <vector> +#include <list> +#include <cstdio> +#include <cstring> +#include <cstdint> +#include <arpa/inet.h> +#include <stdexcept> +#include <sstream> +#include "PFT.h" +#include "crc.h" +#include "ReedSolomon.h" + +namespace edi { + +using namespace std; + +// An integer division that rounds up, i.e. ceil(a/b) +#define CEIL_DIV(a, b) (a % b == 0 ? a / b : a / b + 1) + +PFT::PFT() { } + +PFT::PFT(const configuration_t &conf) : + m_k(conf.chunk_len), + m_m(conf.fec), + m_dest_port(conf.dest_port), + m_pseq(0), + m_num_chunks(0), + m_verbose(conf.verbose) + { + if (m_k > 207) { + etiLog.level(warn) << + "EDI PFT: maximum chunk size is 207."; + throw std::out_of_range("EDI PFT Chunk size too large."); + } + + if (m_m > 5) { + etiLog.level(warn) << + "EDI PFT: high number of recoverable fragments" + " may lead to large overhead"; + // See TS 102 821, 7.2.1 Known values, list entry for 'm' + } + } + +RSBlock PFT::Protect(AFPacket af_packet) +{ + RSBlock rs_block; + + // number of chunks is ceil(afpacketsize / m_k) + // TS 102 821 7.2.2: c = ceil(l / k_max) + m_num_chunks = CEIL_DIV(af_packet.size(), m_k); + + if (m_verbose) { + fprintf(stderr, "Protect %zu chunks of size %zu\n", + m_num_chunks, af_packet.size()); + } + + // calculate size of chunk: + // TS 102 821 7.2.2: k = ceil(l / c) + // chunk_len does not include the 48 bytes of protection. + const size_t chunk_len = CEIL_DIV(af_packet.size(), m_num_chunks); + if (chunk_len > 207) { + std::stringstream ss; + ss << "Chunk length " << chunk_len << " too large (>207)"; + throw std::runtime_error(ss.str()); + } + + // The last RS chunk is zero padded + // TS 102 821 7.2.2: z = c*k - l + const size_t zero_pad = m_num_chunks * chunk_len - af_packet.size(); + + // Create the RS(k+p,k) encoder + const int firstRoot = 1; // Discovered by analysing EDI dump + const int gfPoly = 0x11d; + const bool reverse = false; + // The encoding has to be 255, 207 always, because the chunk has to + // be padded at the end, and not at the beginning as libfec would + // do + ReedSolomon rs_encoder(255, 207, reverse, gfPoly, firstRoot); + + // add zero padding to last chunk + for (size_t i = 0; i < zero_pad; i++) { + af_packet.push_back(0); + } + + if (m_verbose) { + fprintf(stderr, " add %zu zero padding\n", zero_pad); + } + + // Calculate RS for each chunk and assemble RS block + for (size_t i = 0; i < af_packet.size(); i+= chunk_len) { + vector<uint8_t> chunk(207); + vector<uint8_t> protection(PARITYBYTES); + + // copy chunk_len bytes into new chunk + memcpy(&chunk.front(), &af_packet[i], chunk_len); + + // calculate RS for chunk with padding + rs_encoder.encode(&chunk.front(), &protection.front(), 207); + + // Drop the padding + chunk.resize(chunk_len); + + // append new chunk and protection to the RS Packet + rs_block.insert(rs_block.end(), chunk.begin(), chunk.end()); + rs_block.insert(rs_block.end(), protection.begin(), protection.end()); + } + + return rs_block; +} + +vector< vector<uint8_t> > PFT::ProtectAndFragment(AFPacket af_packet) +{ + const bool enable_RS = (m_m > 0); + + if (enable_RS) { + RSBlock rs_block = Protect(af_packet); + +#if 0 + fprintf(stderr, " af_packet (%zu):", af_packet.size()); + for (size_t i = 0; i < af_packet.size(); i++) { + fprintf(stderr, "%02x ", af_packet[i]); + } + fprintf(stderr, "\n"); + + fprintf(stderr, " rs_block (%zu):", rs_block.size()); + for (size_t i = 0; i < rs_block.size(); i++) { + fprintf(stderr, "%02x ", rs_block[i]); + } + fprintf(stderr, "\n"); +#endif + + // TS 102 821 7.2.2: s_max = MIN(floor(c*p/(m+1)), MTU - h)) + const size_t max_payload_size = ( m_num_chunks * PARITYBYTES ) / (m_m + 1); + + // Calculate fragment count and size + // TS 102 821 7.2.2: ceil((l + c*p + z) / s_max) + // l + c*p + z = length of RS block + const size_t num_fragments = CEIL_DIV(rs_block.size(), max_payload_size); + + // TS 102 821 7.2.2: ceil((l + c*p + z) / f) + const size_t fragment_size = CEIL_DIV(rs_block.size(), num_fragments); + + if (m_verbose) + fprintf(stderr, " PnF fragment_size %zu, num frag %zu\n", + fragment_size, num_fragments); + + vector< vector<uint8_t> > fragments(num_fragments); + + for (size_t i = 0; i < num_fragments; i++) { + fragments[i].resize(fragment_size); + for (size_t j = 0; j < fragment_size; j++) { + const size_t ix = j*num_fragments + i; + if (ix < rs_block.size()) { + fragments[i][j] = rs_block[ix]; + } + else { + fragments[i][j] = 0; + } + } + } + + return fragments; + } + else { // No RS, only fragmentation + // TS 102 821 7.2.2: s_max = MTU - h + // Ethernet MTU is 1500, but maybe you are routing over a network which + // has some sort of packet encapsulation. Add some margin. + const size_t max_payload_size = 1400; + + // Calculate fragment count and size + // TS 102 821 7.2.2: ceil((l + c*p + z) / s_max) + // l + c*p + z = length of AF packet + const size_t num_fragments = CEIL_DIV(af_packet.size(), max_payload_size); + + // TS 102 821 7.2.2: ceil((l + c*p + z) / f) + const size_t fragment_size = CEIL_DIV(af_packet.size(), num_fragments); + vector< vector<uint8_t> > fragments(num_fragments); + + for (size_t i = 0; i < num_fragments; i++) { + fragments[i].reserve(fragment_size); + + for (size_t j = 0; j < fragment_size; j++) { + const size_t ix = i*fragment_size + j; + if (ix < af_packet.size()) { + fragments[i].push_back(af_packet.at(ix)); + } + else { + break; + } + } + } + + return fragments; + } +} + +std::vector< PFTFragment > PFT::Assemble(AFPacket af_packet) +{ + vector< vector<uint8_t> > fragments = ProtectAndFragment(af_packet); + vector< vector<uint8_t> > pft_fragments; // These contain PF headers + + const bool enable_RS = (m_m > 0); + const bool enable_transport = true; + + unsigned int findex = 0; + + unsigned fcount = fragments.size(); + + // calculate size of chunk: + // TS 102 821 7.2.2: k = ceil(l / c) + // chunk_len does not include the 48 bytes of protection. + const size_t chunk_len = enable_RS ? + CEIL_DIV(af_packet.size(), m_num_chunks) : 0; + + // The last RS chunk is zero padded + // TS 102 821 7.2.2: z = c*k - l + const size_t zero_pad = enable_RS ? + m_num_chunks * chunk_len - af_packet.size() : 0; + + for (const auto &fragment : fragments) { + // Psync + std::string psync("PF"); + std::vector<uint8_t> packet(psync.begin(), psync.end()); + + // Pseq + packet.push_back(m_pseq >> 8); + packet.push_back(m_pseq & 0xFF); + + // Findex + packet.push_back(findex >> 16); + packet.push_back(findex >> 8); + packet.push_back(findex & 0xFF); + findex++; + + // Fcount + packet.push_back(fcount >> 16); + packet.push_back(fcount >> 8); + packet.push_back(fcount & 0xFF); + + // RS (1 bit), transport (1 bit) and Plen (14 bits) + unsigned int plen = fragment.size(); + if (enable_RS) { + plen |= 0x8000; // Set FEC bit + } + + if (enable_transport) { + plen |= 0x4000; // Set ADDR bit + } + + packet.push_back(plen >> 8); + packet.push_back(plen & 0xFF); + + if (enable_RS) { + packet.push_back(chunk_len); // RSk + packet.push_back(zero_pad); // RSz + } + + if (enable_transport) { + // Source (16 bits) + uint16_t addr_source = 0; + packet.push_back(addr_source >> 8); + packet.push_back(addr_source & 0xFF); + + // Dest (16 bits) + packet.push_back(m_dest_port >> 8); + packet.push_back(m_dest_port & 0xFF); + } + + // calculate CRC over AF Header and payload + uint16_t crc = 0xffff; + crc = crc16(crc, &(packet.front()), packet.size()); + crc ^= 0xffff; + + packet.push_back((crc >> 8) & 0xFF); + packet.push_back(crc & 0xFF); + + // insert payload, must have a length multiple of 8 bytes + packet.insert(packet.end(), fragment.begin(), fragment.end()); + + pft_fragments.push_back(packet); + +#if 0 + fprintf(stderr, "* PFT pseq %d, findex %d, fcount %d, plen %d\n", + m_pseq, findex, fcount, plen & ~0xC000); +#endif + } + + m_pseq++; + + return pft_fragments; +} + +} + diff --git a/lib/edioutput/PFT.h b/lib/edioutput/PFT.h new file mode 100644 index 0000000..502aa39 --- /dev/null +++ b/lib/edioutput/PFT.h @@ -0,0 +1,78 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output, + Protection, Fragmentation and Transport. (PFT) + + Are supported: + Reed-Solomon and Fragmentation + + This implements part of PFT as defined ETSI TS 102 821. + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#include "config.h" +#include <vector> +#include <list> +#include <stdexcept> +#include <cstdint> +#include "AFPacket.h" +#include "Log.h" +#include "ReedSolomon.h" +#include "EDIConfig.h" + +namespace edi { + +typedef std::vector<uint8_t> RSBlock; +typedef std::vector<uint8_t> PFTFragment; + +class PFT +{ + public: + static constexpr int PARITYBYTES = 48; + + PFT(); + PFT(const configuration_t& conf); + + // return a list of PFT fragments with the correct + // PFT headers + std::vector< PFTFragment > Assemble(AFPacket af_packet); + + // Apply Reed-Solomon FEC to the AF Packet + RSBlock Protect(AFPacket af_packet); + + // Cut a RSBlock into several fragments that can be transmitted + std::vector< std::vector<uint8_t> > ProtectAndFragment(AFPacket af_packet); + + private: + unsigned int m_k = 207; // length of RS data word + unsigned int m_m = 3; // number of fragments that can be recovered if lost + unsigned int m_dest_port = 12000; // Destination port for transport header + uint16_t m_pseq = 0; + size_t m_num_chunks = 0; + bool m_verbose = 0; +}; + +} + diff --git a/lib/edioutput/TagItems.cpp b/lib/edioutput/TagItems.cpp new file mode 100644 index 0000000..9746469 --- /dev/null +++ b/lib/edioutput/TagItems.cpp @@ -0,0 +1,449 @@ +/* + EDI output. + This defines a few TAG items as defined ETSI TS 102 821 and + ETSI TS 102 693 + + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include "config.h" +#include "TagItems.h" +#include <vector> +#include <iostream> +#include <string> +#include <cstdint> +#include <stdexcept> + +namespace edi { + +TagStarPTR::TagStarPTR(const std::string& protocol) + : m_protocol(protocol) +{ + if (m_protocol.size() != 4) { + throw std::runtime_error("TagStarPTR protocol invalid length"); + } +} + +std::vector<uint8_t> TagStarPTR::Assemble() +{ + //std::cerr << "TagItem *ptr" << std::endl; + std::string pack_data("*ptr"); + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + packet.push_back(0x40); + + packet.insert(packet.end(), m_protocol.begin(), m_protocol.end()); + + // Major + packet.push_back(0); + packet.push_back(0); + + // Minor + packet.push_back(0); + packet.push_back(0); + return packet; +} + +std::vector<uint8_t> TagDETI::Assemble() +{ + std::string pack_data("deti"); + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + packet.reserve(256); + + // Placeholder for length + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + + uint8_t fct = dlfc % 250; + uint8_t fcth = dlfc / 250; + + + uint16_t detiHeader = fct | (fcth << 8) | (rfudf << 13) | (ficf << 14) | (atstf << 15); + packet.push_back(detiHeader >> 8); + packet.push_back(detiHeader & 0xFF); + + uint32_t etiHeader = mnsc | (rfu << 16) | (rfa << 17) | + (fp << 19) | (mid << 22) | (stat << 24); + packet.push_back((etiHeader >> 24) & 0xFF); + packet.push_back((etiHeader >> 16) & 0xFF); + packet.push_back((etiHeader >> 8) & 0xFF); + packet.push_back(etiHeader & 0xFF); + + if (atstf) { + packet.push_back(utco); + + packet.push_back((seconds >> 24) & 0xFF); + packet.push_back((seconds >> 16) & 0xFF); + packet.push_back((seconds >> 8) & 0xFF); + packet.push_back(seconds & 0xFF); + + packet.push_back((tsta >> 16) & 0xFF); + packet.push_back((tsta >> 8) & 0xFF); + packet.push_back(tsta & 0xFF); + } + + if (ficf) { + for (size_t i = 0; i < fic_length; i++) { + packet.push_back(fic_data[i]); + } + } + + if (rfudf) { + packet.push_back((rfud >> 16) & 0xFF); + packet.push_back((rfud >> 8) & 0xFF); + packet.push_back(rfud & 0xFF); + } + + // calculate and update size + // remove TAG name and TAG length fields and convert to bits + uint32_t taglength = (packet.size() - 8) * 8; + + // write length into packet + packet[4] = (taglength >> 24) & 0xFF; + packet[5] = (taglength >> 16) & 0xFF; + packet[6] = (taglength >> 8) & 0xFF; + packet[7] = taglength & 0xFF; + + dlfc = (dlfc+1) % 5000; + + /* + std::cerr << "TagItem deti, packet.size " << packet.size() << std::endl; + std::cerr << " fic length " << fic_length << std::endl; + std::cerr << " length " << taglength / 8 << std::endl; + */ + return packet; +} + +void TagDETI::set_edi_time(const std::time_t t, int tai_utc_offset) +{ + utco = tai_utc_offset - 32; + + const std::time_t posix_timestamp_1_jan_2000 = 946684800; + + seconds = t - posix_timestamp_1_jan_2000 + utco; +} + +std::vector<uint8_t> TagESTn::Assemble() +{ + std::string pack_data("est"); + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + packet.reserve(mst_length*8 + 16); + + packet.push_back(id); + + // Placeholder for length + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + + if (tpl > 0x3F) { + throw std::runtime_error("TagESTn: invalid TPL value"); + } + + if (sad > 0x3FF) { + throw std::runtime_error("TagESTn: invalid SAD value"); + } + + if (scid > 0x3F) { + throw std::runtime_error("TagESTn: invalid SCID value"); + } + + uint32_t sstc = (scid << 18) | (sad << 8) | (tpl << 2) | rfa; + packet.push_back((sstc >> 16) & 0xFF); + packet.push_back((sstc >> 8) & 0xFF); + packet.push_back(sstc & 0xFF); + + for (size_t i = 0; i < mst_length * 8; i++) { + packet.push_back(mst_data[i]); + } + + // calculate and update size + // remove TAG name and TAG length fields and convert to bits + uint32_t taglength = (packet.size() - 8) * 8; + + // write length into packet + packet[4] = (taglength >> 24) & 0xFF; + packet[5] = (taglength >> 16) & 0xFF; + packet[6] = (taglength >> 8) & 0xFF; + packet[7] = taglength & 0xFF; + + /* + std::cerr << "TagItem ESTn, length " << packet.size() << std::endl; + std::cerr << " mst_length " << mst_length << std::endl; + */ + return packet; +} + +std::vector<uint8_t> TagDSTI::Assemble() +{ + std::string pack_data("dsti"); + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + packet.reserve(256); + + // Placeholder for length + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + + uint8_t dfctl = dflc % 250; + uint8_t dfcth = dflc / 250; + + + uint16_t dstiHeader = dfctl | (dfcth << 8) | (rfadf << 13) | (atstf << 14) | (stihf << 15); + packet.push_back(dstiHeader >> 8); + packet.push_back(dstiHeader & 0xFF); + + if (stihf) { + packet.push_back(stat); + packet.push_back((spid >> 8) & 0xFF); + packet.push_back(spid & 0xFF); + } + + if (atstf) { + packet.push_back(utco); + + packet.push_back((seconds >> 24) & 0xFF); + packet.push_back((seconds >> 16) & 0xFF); + packet.push_back((seconds >> 8) & 0xFF); + packet.push_back(seconds & 0xFF); + + packet.push_back((tsta >> 16) & 0xFF); + packet.push_back((tsta >> 8) & 0xFF); + packet.push_back(tsta & 0xFF); + } + + if (rfadf) { + for (size_t i = 0; i < rfad.size(); i++) { + packet.push_back(rfad[i]); + } + } + // calculate and update size + // remove TAG name and TAG length fields and convert to bits + uint32_t taglength = (packet.size() - 8) * 8; + + // write length into packet + packet[4] = (taglength >> 24) & 0xFF; + packet[5] = (taglength >> 16) & 0xFF; + packet[6] = (taglength >> 8) & 0xFF; + packet[7] = taglength & 0xFF; + + dflc = (dflc+1) % 5000; + + /* + std::cerr << "TagItem dsti, packet.size " << packet.size() << std::endl; + std::cerr << " length " << taglength / 8 << std::endl; + */ + return packet; +} + +void TagDSTI::set_edi_time(const std::time_t t, int tai_utc_offset) +{ + utco = tai_utc_offset - 32; + + const std::time_t posix_timestamp_1_jan_2000 = 946684800; + + seconds = t - posix_timestamp_1_jan_2000 + utco; +} + +#if 0 +/* Update the EDI time. t is in UTC, TAI offset is requested from adjtimex */ +void TagDSTI::set_edi_time(const std::time_t t) +{ + if (tai_offset_cache_updated_at == 0 or tai_offset_cache_updated_at + 3600 < t) { + struct timex timex_request; + timex_request.modes = 0; + + int err = adjtimex(&timex_request); + if (err == -1) { + throw std::runtime_error("adjtimex failed"); + } + + if (timex_request.tai == 0) { + throw std::runtime_error("CLOCK_TAI is not properly set up"); + } + tai_offset_cache = timex_request.tai; + tai_offset_cache_updated_at = t; + + fprintf(stderr, "adjtimex: %d, tai %d\n", err, timex_request.tai); + } + + utco = tai_offset_cache - 32; + + const std::time_t posix_timestamp_1_jan_2000 = 946684800; + + seconds = t - posix_timestamp_1_jan_2000 + utco; +} +#endif + +std::vector<uint8_t> TagSSm::Assemble() +{ + std::string pack_data("ss"); + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + packet.reserve(istd_length + 16); + + packet.push_back((id >> 8) & 0xFF); + packet.push_back(id & 0xFF); + + // Placeholder for length + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + packet.push_back(0); + + if (rfa > 0x1F) { + throw std::runtime_error("TagSSm: invalid RFA value"); + } + + if (tid > 0x7) { + throw std::runtime_error("TagSSm: invalid tid value"); + } + + if (tidext > 0x7) { + throw std::runtime_error("TagSSm: invalid tidext value"); + } + + if (stid > 0x0FFF) { + throw std::runtime_error("TagSSm: invalid stid value"); + } + + uint32_t istc = (rfa << 19) | (tid << 16) | (tidext << 13) | ((crcstf ? 1 : 0) << 12) | stid; + packet.push_back((istc >> 16) & 0xFF); + packet.push_back((istc >> 8) & 0xFF); + packet.push_back(istc & 0xFF); + + for (size_t i = 0; i < istd_length; i++) { + packet.push_back(istd_data[i]); + } + + // calculate and update size + // remove TAG name and TAG length fields and convert to bits + uint32_t taglength = (packet.size() - 8) * 8; + + // write length into packet + packet[4] = (taglength >> 24) & 0xFF; + packet[5] = (taglength >> 16) & 0xFF; + packet[6] = (taglength >> 8) & 0xFF; + packet[7] = taglength & 0xFF; + + /* + std::cerr << "TagItem SSm, length " << packet.size() << std::endl; + std::cerr << " istd_length " << istd_length << std::endl; + */ + return packet; +} + + +std::vector<uint8_t> TagStarDMY::Assemble() +{ + std::string pack_data("*dmy"); + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + + packet.resize(4 + 4 + length_); + + const uint32_t length_bits = length_ * 8; + + packet[4] = (length_bits >> 24) & 0xFF; + packet[5] = (length_bits >> 16) & 0xFF; + packet[6] = (length_bits >> 8) & 0xFF; + packet[7] = length_bits & 0xFF; + + // The remaining bytes in the packet are "undefined data" + + return packet; +} + +TagODRVersion::TagODRVersion(const std::string& version, uint32_t uptime_s) : + m_version(version), + m_uptime(uptime_s) +{ +} + +std::vector<uint8_t> TagODRVersion::Assemble() +{ + std::string pack_data("ODRv"); + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + + const size_t length = m_version.size() + sizeof(uint32_t); + + packet.resize(4 + 4 + length); + + const uint32_t length_bits = length * 8; + + size_t i = 4; + packet[i++] = (length_bits >> 24) & 0xFF; + packet[i++] = (length_bits >> 16) & 0xFF; + packet[i++] = (length_bits >> 8) & 0xFF; + packet[i++] = length_bits & 0xFF; + + copy(m_version.cbegin(), m_version.cend(), packet.begin() + i); + i += m_version.size(); + + packet[i++] = (m_uptime >> 24) & 0xFF; + packet[i++] = (m_uptime >> 16) & 0xFF; + packet[i++] = (m_uptime >> 8) & 0xFF; + packet[i++] = m_uptime & 0xFF; + + return packet; +} + +TagODRAudioLevels::TagODRAudioLevels(int16_t audiolevel_left, int16_t audiolevel_right) : + m_audio_left(audiolevel_left), + m_audio_right(audiolevel_right) +{ +} + +std::vector<uint8_t> TagODRAudioLevels::Assemble() +{ + std::string pack_data("ODRa"); + std::vector<uint8_t> packet(pack_data.begin(), pack_data.end()); + + constexpr size_t length = 2*sizeof(int16_t); + + packet.resize(4 + 4 + length); + + const uint32_t length_bits = length * 8; + + size_t i = 4; + packet[i++] = (length_bits >> 24) & 0xFF; + packet[i++] = (length_bits >> 16) & 0xFF; + packet[i++] = (length_bits >> 8) & 0xFF; + packet[i++] = length_bits & 0xFF; + + packet[i++] = (m_audio_left >> 8) & 0xFF; + packet[i++] = m_audio_left & 0xFF; + + packet[i++] = (m_audio_right >> 8) & 0xFF; + packet[i++] = m_audio_right & 0xFF; + + return packet; +} + +} + diff --git a/lib/edioutput/TagItems.h b/lib/edioutput/TagItems.h new file mode 100644 index 0000000..5c81b01 --- /dev/null +++ b/lib/edioutput/TagItems.h @@ -0,0 +1,253 @@ +/* + EDI output. + This defines a few TAG items as defined ETSI TS 102 821 and + ETSI TS 102 693 + + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#include "config.h" +#include <vector> +#include <array> +#include <chrono> +#include <string> +#include <cstdint> + +namespace edi { + +class TagItem +{ + public: + virtual std::vector<uint8_t> Assemble() = 0; +}; + +// ETSI TS 102 693, 5.1.1 Protocol type and revision +class TagStarPTR : public TagItem +{ + public: + TagStarPTR(const std::string& protocol); + std::vector<uint8_t> Assemble(); + + private: + std::string m_protocol = ""; +}; + +// ETSI TS 102 693, 5.1.3 DAB ETI(LI) Management (deti) +class TagDETI : public TagItem +{ + public: + std::vector<uint8_t> Assemble(); + + /***** DATA in intermediary format ****/ + // For the ETI Header: must be defined ! + uint8_t stat = 0; + uint8_t mid = 0; + uint8_t fp = 0; + uint8_t rfa = 0; + uint8_t rfu = 0; // MNSC is valid + uint16_t mnsc = 0; + uint16_t dlfc = 0; // modulo 5000 frame counter + + // ATST (optional) + bool atstf = false; // presence of atst data + + /* UTCO: Offset (in seconds) between UTC and the Seconds value. The + * value is expressed as an unsigned 8-bit quantity. As of February + * 2009, the value shall be 2 and shall change as a result of each + * modification of the number of leap seconds, as proscribed by + * International Earth Rotation and Reference Systems Service (IERS). + * + * According to Annex F + * EDI = TAI - 32s (constant) + * EDI = UTC + UTCO + * we derive + * UTCO = TAI-UTC - 32 + * where the TAI-UTC offset is given by the USNO bulletin using + * the ClockTAI module. + */ + uint8_t utco = 0; + + /* Update the EDI time. t is in UTC */ + void set_edi_time(const std::time_t t, int tai_utc_offset); + + /* The number of SI seconds since 2000-01-01 T 00:00:00 UTC as an + * unsigned 32-bit quantity. Contrary to POSIX, this value also + * counts leap seconds. + */ + uint32_t seconds = 0; + + /* TSTA: Shall be the 24 least significant bits of the Time Stamp + * (TIST) field from the STI-D(LI) Frame. The full definition for the + * STI TIST can be found in annex B of EN 300 797 [4]. The most + * significant 8 bits of the TIST field of the incoming STI-D(LI) + * frame, if required, may be carried in the RFAD field. + */ + uint32_t tsta = 0xFFFFFF; + + // the FIC (optional) + bool ficf = false; + const unsigned char* fic_data; + size_t fic_length; + + // rfu + bool rfudf = false; + uint32_t rfud = 0; + + +}; + +// ETSI TS 102 693, 5.1.5 ETI Sub-Channel Stream <n> +class TagESTn : public TagItem +{ + public: + std::vector<uint8_t> Assemble(); + + // SSTCn + uint8_t scid; + uint16_t sad; + uint8_t tpl; + uint8_t rfa; + + // Pointer to MSTn data + uint8_t* mst_data; + size_t mst_length; // STLn * 8 bytes + + uint8_t id; +}; + +// ETSI TS 102 693, 5.1.2 DAB STI-D(LI) Management +class TagDSTI : public TagItem +{ + public: + std::vector<uint8_t> Assemble(); + + // dsti Header + bool stihf = false; + bool atstf = false; // presence of atst data + bool rfadf = false; + uint16_t dflc = 0; // modulo 5000 frame counter + + // STI Header (optional) + uint8_t stat = 0; + uint16_t spid = 0; + + /* UTCO: Offset (in seconds) between UTC and the Seconds value. The + * value is expressed as an unsigned 8-bit quantity. As of February + * 2009, the value shall be 2 and shall change as a result of each + * modification of the number of leap seconds, as proscribed by + * International Earth Rotation and Reference Systems Service (IERS). + * + * According to Annex F + * EDI = TAI - 32s (constant) + * EDI = UTC + UTCO + * we derive + * UTCO = TAI-UTC - 32 + * where the TAI-UTC offset is given by the USNO bulletin using + * the ClockTAI module. + */ + uint8_t utco = 0; + + /* Update the EDI time. t is in UTC */ + void set_edi_time(const std::time_t t, int tai_utc_offset); + + /* The number of SI seconds since 2000-01-01 T 00:00:00 UTC as an + * unsigned 32-bit quantity. Contrary to POSIX, this value also + * counts leap seconds. + */ + uint32_t seconds = 0; + + /* TSTA: Shall be the 24 least significant bits of the Time Stamp + * (TIST) field from the STI-D(LI) Frame. The full definition for the + * STI TIST can be found in annex B of EN 300 797 [4]. The most + * significant 8 bits of the TIST field of the incoming STI-D(LI) + * frame, if required, may be carried in the RFAD field. + */ + uint32_t tsta = 0xFFFFFF; + + std::array<uint8_t, 9> rfad; + + private: + int tai_offset_cache = 0; + std::time_t tai_offset_cache_updated_at = 0; +}; + +// ETSI TS 102 693, 5.1.4 STI-D Payload Stream <m> +class TagSSm : public TagItem +{ + public: + std::vector<uint8_t> Assemble(); + + // SSTCn + uint8_t rfa = 0; + uint8_t tid = 0; // See EN 300 797, 5.4.1.1. Value 0 means "MSC sub-channel" + uint8_t tidext = 0; // EN 300 797, 5.4.1.3, Value 0 means "MSC audio stream" + bool crcstf = false; + uint16_t stid = 0; + + // Pointer to ISTDm data + const uint8_t *istd_data; + size_t istd_length; // bytes + + uint16_t id = 0; +}; + +// ETSI TS 102 821, 5.2.2.2 Dummy padding +class TagStarDMY : public TagItem +{ + public: + /* length is the TAG value length in bytes */ + TagStarDMY(uint32_t length) : length_(length) {} + std::vector<uint8_t> Assemble(); + + private: + uint32_t length_; +}; + +// Custom TAG that carries version information of the EDI source +class TagODRVersion : public TagItem +{ + public: + TagODRVersion(const std::string& version, uint32_t uptime_s); + std::vector<uint8_t> Assemble(); + + private: + std::string m_version; + uint32_t m_uptime; +}; + +// Custom TAG that carries audio level metadata +class TagODRAudioLevels : public TagItem +{ + public: + TagODRAudioLevels(int16_t audiolevel_left, int16_t audiolevel_right); + std::vector<uint8_t> Assemble(); + + private: + int16_t m_audio_left; + int16_t m_audio_right; +}; + +} + diff --git a/lib/edioutput/TagPacket.cpp b/lib/edioutput/TagPacket.cpp new file mode 100644 index 0000000..b0bf9a1 --- /dev/null +++ b/lib/edioutput/TagPacket.cpp @@ -0,0 +1,78 @@ +/* + Copyright (C) 2014 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output. + This defines a TAG Packet. + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include "config.h" +#include "TagPacket.h" +#include "TagItems.h" +#include <vector> +#include <iostream> +#include <string> +#include <list> +#include <cstdint> +#include <cassert> + +namespace edi { + +TagPacket::TagPacket(unsigned int alignment) : m_alignment(alignment) +{ } + +std::vector<uint8_t> TagPacket::Assemble() +{ + std::list<TagItem*>::iterator tag; + + std::vector<uint8_t> packet; + + //std::cerr << "Assemble TAGPacket" << std::endl; + + for (tag = tag_items.begin(); tag != tag_items.end(); ++tag) { + std::vector<uint8_t> tag_data = (*tag)->Assemble(); + packet.insert(packet.end(), tag_data.begin(), tag_data.end()); + + //std::cerr << " Add TAGItem of length " << tag_data.size() << std::endl; + } + + if (m_alignment == 0) { /* no padding */ } + else if (m_alignment == 8) { + // Add padding inside TAG packet + while (packet.size() % 8 > 0) { + packet.push_back(0); // TS 102 821, 5.1, "padding shall be undefined" + } + } + else if (m_alignment > 8) { + TagStarDMY dmy(m_alignment - 8); + auto dmy_data = dmy.Assemble(); + packet.insert(packet.end(), dmy_data.begin(), dmy_data.end()); + } + else { + std::cerr << "Invalid alignment requirement " << m_alignment << + " defined in TagPacket" << std::endl; + } + + return packet; +} + +} + diff --git a/lib/edioutput/TagPacket.h b/lib/edioutput/TagPacket.h new file mode 100644 index 0000000..1e40ce7 --- /dev/null +++ b/lib/edioutput/TagPacket.h @@ -0,0 +1,56 @@ +/* + Copyright (C) 2014 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output. + This defines a TAG Packet. + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#include "config.h" +#include "TagItems.h" +#include <vector> +#include <string> +#include <list> +#include <cstdint> + +namespace edi { + +// A TagPacket is nothing else than a list of tag items, with an +// Assemble function that puts the bytestream together and adds +// padding such that the total length is a multiple of 8 Bytes. +// +// ETSI TS 102 821, 5.1 Tag Packet +class TagPacket +{ + public: + TagPacket(unsigned int alignment); + std::vector<uint8_t> Assemble(); + + std::list<TagItem*> tag_items; + + private: + unsigned int m_alignment; +}; + +} + diff --git a/lib/edioutput/Transport.cpp b/lib/edioutput/Transport.cpp new file mode 100644 index 0000000..4c91483 --- /dev/null +++ b/lib/edioutput/Transport.cpp @@ -0,0 +1,187 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output, + UDP and TCP transports and their configuration + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ +#include "Transport.h" +#include <iterator> + +using namespace std; + +namespace edi { + +void configuration_t::print() const +{ + etiLog.level(info) << "EDI"; + etiLog.level(info) << " verbose " << verbose; + for (auto edi_dest : destinations) { + if (auto udp_dest = dynamic_pointer_cast<edi::udp_destination_t>(edi_dest)) { + etiLog.level(info) << " UDP to " << udp_dest->dest_addr << ":" << dest_port; + if (not udp_dest->source_addr.empty()) { + etiLog.level(info) << " source " << udp_dest->source_addr; + etiLog.level(info) << " ttl " << udp_dest->ttl; + } + etiLog.level(info) << " source port " << udp_dest->source_port; + } + else if (auto tcp_dest = dynamic_pointer_cast<edi::tcp_server_t>(edi_dest)) { + etiLog.level(info) << " TCP listening on port " << tcp_dest->listen_port; + etiLog.level(info) << " max frames queued " << tcp_dest->max_frames_queued; + } + else if (auto tcp_dest = dynamic_pointer_cast<edi::tcp_client_t>(edi_dest)) { + etiLog.level(info) << " TCP client connecting to " << tcp_dest->dest_addr << ":" << tcp_dest->dest_port; + etiLog.level(info) << " max frames queued " << tcp_dest->max_frames_queued; + } + else { + throw logic_error("EDI destination not implemented"); + } + } + if (interleaver_enabled()) { + etiLog.level(info) << " interleave " << latency_frames * 24 << " ms"; + } +} + + +Sender::Sender(const configuration_t& conf) : + m_conf(conf), + edi_pft(m_conf) +{ + if (m_conf.verbose) { + etiLog.log(info, "Setup EDI"); + } + + for (const auto& edi_dest : m_conf.destinations) { + if (const auto udp_dest = dynamic_pointer_cast<edi::udp_destination_t>(edi_dest)) { + auto udp_socket = std::make_shared<Socket::UDPSocket>(udp_dest->source_port); + + if (not udp_dest->source_addr.empty()) { + udp_socket->setMulticastSource(udp_dest->source_addr.c_str()); + udp_socket->setMulticastTTL(udp_dest->ttl); + } + + udp_sockets.emplace(udp_dest.get(), udp_socket); + } + else if (auto tcp_dest = dynamic_pointer_cast<edi::tcp_server_t>(edi_dest)) { + auto dispatcher = make_shared<Socket::TCPDataDispatcher>(tcp_dest->max_frames_queued); + dispatcher->start(tcp_dest->listen_port, "0.0.0.0"); + tcp_dispatchers.emplace(tcp_dest.get(), dispatcher); + } + else if (auto tcp_dest = dynamic_pointer_cast<edi::tcp_client_t>(edi_dest)) { + auto tcp_send_client = make_shared<Socket::TCPSendClient>(tcp_dest->dest_addr, tcp_dest->dest_port); + tcp_senders.emplace(tcp_dest.get(), tcp_send_client); + } + else { + throw logic_error("EDI destination not implemented"); + } + } + + if (m_conf.interleaver_enabled()) { + edi_interleaver.SetLatency(m_conf.latency_frames); + } + + if (m_conf.dump) { + edi_debug_file.open("./edi.debug"); + } + + if (m_conf.verbose) { + etiLog.log(info, "EDI set up"); + } +} + +void Sender::write(const TagPacket& tagpacket) +{ + // Assemble into one AF Packet + edi::AFPacket af_packet = edi_afPacketiser.Assemble(tagpacket); + + if (m_conf.enable_pft) { + // Apply PFT layer to AF Packet (Reed Solomon FEC and Fragmentation) + vector<edi::PFTFragment> edi_fragments = edi_pft.Assemble(af_packet); + + if (m_conf.verbose) { + fprintf(stderr, "EDI number of PFT fragment before interleaver %zu\n", + edi_fragments.size()); + } + + if (m_conf.interleaver_enabled()) { + edi_fragments = edi_interleaver.Interleave(edi_fragments); + } + + if (m_conf.verbose) { + fprintf(stderr, "EDI number of PFT fragments %zu\n", + edi_fragments.size()); + } + + // Send over ethernet + for (auto& edi_frag : edi_fragments) { + if (m_conf.dump) { + ostream_iterator<uint8_t> debug_iterator(edi_debug_file); + copy(edi_frag.begin(), edi_frag.end(), debug_iterator); + } + + for (auto& dest : m_conf.destinations) { + if (const auto& udp_dest = dynamic_pointer_cast<edi::udp_destination_t>(dest)) { + Socket::InetAddress addr; + addr.resolveUdpDestination(udp_dest->dest_addr, m_conf.dest_port); + + udp_sockets.at(udp_dest.get())->send(edi_frag, addr); + } + else if (auto tcp_dest = dynamic_pointer_cast<edi::tcp_server_t>(dest)) { + tcp_dispatchers.at(tcp_dest.get())->write(edi_frag); + } + else if (auto tcp_dest = dynamic_pointer_cast<edi::tcp_client_t>(dest)) { + tcp_senders.at(tcp_dest.get())->sendall(edi_frag); + } + else { + throw logic_error("EDI destination not implemented"); + } + } + } + } + else { + // Send over ethernet + if (m_conf.dump) { + ostream_iterator<uint8_t> debug_iterator(edi_debug_file); + copy(af_packet.begin(), af_packet.end(), debug_iterator); + } + + for (auto& dest : m_conf.destinations) { + if (const auto& udp_dest = dynamic_pointer_cast<edi::udp_destination_t>(dest)) { + Socket::InetAddress addr; + addr.resolveUdpDestination(udp_dest->dest_addr, m_conf.dest_port); + + udp_sockets.at(udp_dest.get())->send(af_packet, addr); + } + else if (auto tcp_dest = dynamic_pointer_cast<edi::tcp_server_t>(dest)) { + tcp_dispatchers.at(tcp_dest.get())->write(af_packet); + } + else if (auto tcp_dest = dynamic_pointer_cast<edi::tcp_client_t>(dest)) { + tcp_senders.at(tcp_dest.get())->sendall(af_packet); + } + else { + throw logic_error("EDI destination not implemented"); + } + } + } +} + +} diff --git a/lib/edioutput/Transport.h b/lib/edioutput/Transport.h new file mode 100644 index 0000000..73b2ab6 --- /dev/null +++ b/lib/edioutput/Transport.h @@ -0,0 +1,71 @@ +/* + Copyright (C) 2019 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + + EDI output, + UDP and TCP transports and their configuration + + */ +/* + This file is part of the ODR-mmbTools. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#pragma once + +#include "config.h" +#include "EDIConfig.h" +#include "AFPacket.h" +#include "PFT.h" +#include "Interleaver.h" +#include "Socket.h" +#include <vector> +#include <unordered_map> +#include <stdexcept> +#include <fstream> +#include <cstdint> + +namespace edi { + +/** Configuration for EDI output */ + +class Sender { + public: + Sender(const configuration_t& conf); + + void write(const TagPacket& tagpacket); + + private: + configuration_t m_conf; + std::ofstream edi_debug_file; + + // The TagPacket will then be placed into an AFPacket + edi::AFPacketiser edi_afPacketiser; + + // The AF Packet will be protected with reed-solomon and split in fragments + edi::PFT edi_pft; + + // To mitigate for burst packet loss, PFT fragments can be sent out-of-order + edi::Interleaver edi_interleaver; + + std::unordered_map<udp_destination_t*, std::shared_ptr<Socket::UDPSocket>> udp_sockets; + std::unordered_map<tcp_server_t*, std::shared_ptr<Socket::TCPDataDispatcher>> tcp_dispatchers; + std::unordered_map<tcp_client_t*, std::shared_ptr<Socket::TCPSendClient>> tcp_senders; +}; + +} + diff --git a/lib/fec/LICENSE b/lib/fec/LICENSE new file mode 100644 index 0000000..5a883d3 --- /dev/null +++ b/lib/fec/LICENSE @@ -0,0 +1,502 @@ +GNU LESSER GENERAL PUBLIC LICENSE + Version 2.1, February 1999 + + Copyright (C) 1991, 1999 Free Software Foundation, Inc. + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +(This is the first released version of the Lesser GPL. It also counts + as the successor of the GNU Library Public License, version 2, hence + the version number 2.1.) + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Lesser General Public License, applies to some +specially designated software packages--typically libraries--of the +Free Software Foundation and other authors who decide to use it. You +can use it too, but we suggest you first think carefully about whether +this license or the ordinary General Public License is the better +strategy to use in any particular case, based on the explanations below. + + When we speak of free software, we are referring to freedom of use, +not price. Our General Public Licenses are designed to make sure that +you have the freedom to distribute copies of free software (and charge +for this service if you wish); that you receive source code or can get +it if you want it; that you can change the software and use pieces of +it in new free programs; and that you are informed that you can do +these things. + + To protect your rights, we need to make restrictions that forbid +distributors to deny you these rights or to ask you to surrender these +rights. These restrictions translate to certain responsibilities for +you if you distribute copies of the library or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link other code with the library, you must provide +complete object files to the recipients, so that they can relink them +with the library after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + We protect your rights with a two-step method: (1) we copyright the +library, and (2) we offer you this license, which gives you legal +permission to copy, distribute and/or modify the library. + + To protect each distributor, we want to make it very clear that +there is no warranty for the free library. Also, if the library is +modified by someone else and passed on, the recipients should know +that what they have is not the original version, so that the original +author's reputation will not be affected by problems that might be +introduced by others. + + Finally, software patents pose a constant threat to the existence of +any free program. We wish to make sure that a company cannot +effectively restrict the users of a free program by obtaining a +restrictive license from a patent holder. Therefore, we insist that +any patent license obtained for a version of the library must be +consistent with the full freedom of use specified in this license. + + Most GNU software, including some libraries, is covered by the +ordinary GNU General Public License. This license, the GNU Lesser +General Public License, applies to certain designated libraries, and +is quite different from the ordinary General Public License. We use +this license for certain libraries in order to permit linking those +libraries into non-free programs. + + When a program is linked with a library, whether statically or using +a shared library, the combination of the two is legally speaking a +combined work, a derivative of the original library. The ordinary +General Public License therefore permits such linking only if the +entire combination fits its criteria of freedom. The Lesser General +Public License permits more lax criteria for linking other code with +the library. + + We call this license the "Lesser" General Public License because it +does Less to protect the user's freedom than the ordinary General +Public License. It also provides other free software developers Less +of an advantage over competing non-free programs. These disadvantages +are the reason we use the ordinary General Public License for many +libraries. However, the Lesser license provides advantages in certain +special circumstances. + + For example, on rare occasions, there may be a special need to +encourage the widest possible use of a certain library, so that it becomes +a de-facto standard. To achieve this, non-free programs must be +allowed to use the library. A more frequent case is that a free +library does the same job as widely used non-free libraries. In this +case, there is little to gain by limiting the free library to free +software only, so we use the Lesser General Public License. + + In other cases, permission to use a particular library in non-free +programs enables a greater number of people to use a large body of +free software. For example, permission to use the GNU C Library in +non-free programs enables many more people to use the whole GNU +operating system, as well as its variant, the GNU/Linux operating +system. + + Although the Lesser General Public License is Less protective of the +users' freedom, it does ensure that the user of a program that is +linked with the Library has the freedom and the wherewithal to run +that program using a modified version of the Library. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, whereas the latter must +be combined with the library in order to run. + + GNU LESSER GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library or other +program which contains a notice placed by the copyright holder or +other authorized party saying it may be distributed under the terms of +this Lesser General Public License (also called "this License"). +Each licensee is addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control compilation +and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also combine or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (1) uses at run time a + copy of the library already present on the user's computer system, + rather than copying library functions into the executable, and (2) + will operate properly with a modified version of the library, if + the user installs one, as long as the modified version is + interface-compatible with the version that the work was made with. + + c) Accompany the work with a written offer, valid for at + least three years, to give the same user the materials + specified in Subsection 6a, above, for a charge no more + than the cost of performing this distribution. + + d) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + e) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the materials to be distributed need not include anything that is +normally distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties with +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any +particular circumstance, the balance of the section is intended to apply, +and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License may add +an explicit geographical distribution limitation excluding those countries, +so that distribution is permitted only in or among countries not thus +excluded. In such case, this License incorporates the limitation as if +written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Lesser General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Libraries + + If you develop a new library, and you want it to be of the greatest +possible use to the public, we recommend making it free software that +everyone can redistribute and change. You can do so by permitting +redistribution under these terms (or, alternatively, under the terms of the +ordinary General Public License). + + To apply these terms, attach the following notices to the library. It is +safest to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + {description} + Copyright (C) {year} {fullname} + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +Also add information on how to contact you by electronic and paper mail. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the library, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the + library `Frob' (a library for tweaking knobs) written by James Random Hacker. + + {signature of Ty Coon}, 1 April 1990 + Ty Coon, President of Vice + +That's all there is to it! diff --git a/lib/fec/README.md b/lib/fec/README.md new file mode 100644 index 0000000..a44d28d --- /dev/null +++ b/lib/fec/README.md @@ -0,0 +1,12 @@ +FEC routines from KA9Q's libfec +=============================== + +This folder contains part of the libfec library by KA9Q. Only the +char-sized Reed-Solomon encoder and decoder is here. + +The files have been copied from the libfec fork at +https://github.com/Opendigitalradio/ka9q-fec + +Original code is at http://www.ka9q.net/code/fec/ + +All files in this folder are licenced under the LGPL v2.1, please see LICENCE diff --git a/lib/fec/char.h b/lib/fec/char.h new file mode 100644 index 0000000..25efd65 --- /dev/null +++ b/lib/fec/char.h @@ -0,0 +1,24 @@ +/* Stuff specific to the 8-bit symbol version of the general purpose RS codecs + * + * Copyright 2003, Phil Karn, KA9Q + * May be used under the terms of the GNU Lesser General Public License (LGPL) + */ +typedef unsigned char data_t; + +#define MODNN(x) modnn(rs,x) + +#define MM (rs->mm) +#define NN (rs->nn) +#define ALPHA_TO (rs->alpha_to) +#define INDEX_OF (rs->index_of) +#define GENPOLY (rs->genpoly) +#define NROOTS (rs->nroots) +#define FCR (rs->fcr) +#define PRIM (rs->prim) +#define IPRIM (rs->iprim) +#define PAD (rs->pad) +#define A0 (NN) + + + + diff --git a/lib/fec/decode_rs.h b/lib/fec/decode_rs.h new file mode 100644 index 0000000..c165cf3 --- /dev/null +++ b/lib/fec/decode_rs.h @@ -0,0 +1,298 @@ +/* The guts of the Reed-Solomon decoder, meant to be #included + * into a function body with the following typedefs, macros and variables supplied + * according to the code parameters: + + * data_t - a typedef for the data symbol + * data_t data[] - array of NN data and parity symbols to be corrected in place + * retval - an integer lvalue into which the decoder's return code is written + * NROOTS - the number of roots in the RS code generator polynomial, + * which is the same as the number of parity symbols in a block. + Integer variable or literal. + * NN - the total number of symbols in a RS block. Integer variable or literal. + * PAD - the number of pad symbols in a block. Integer variable or literal. + * ALPHA_TO - The address of an array of NN elements to convert Galois field + * elements in index (log) form to polynomial form. Read only. + * INDEX_OF - The address of an array of NN elements to convert Galois field + * elements in polynomial form to index (log) form. Read only. + * MODNN - a function to reduce its argument modulo NN. May be inline or a macro. + * FCR - An integer literal or variable specifying the first consecutive root of the + * Reed-Solomon generator polynomial. Integer variable or literal. + * PRIM - The primitive root of the generator poly. Integer variable or literal. + * DEBUG - If set to 1 or more, do various internal consistency checking. Leave this + * undefined for production code + + * The memset(), memmove(), and memcpy() functions are used. The appropriate header + * file declaring these functions (usually <string.h>) must be included by the calling + * program. + */ + + +#if !defined(NROOTS) +#error "NROOTS not defined" +#endif + +#if !defined(NN) +#error "NN not defined" +#endif + +#if !defined(PAD) +#error "PAD not defined" +#endif + +#if !defined(ALPHA_TO) +#error "ALPHA_TO not defined" +#endif + +#if !defined(INDEX_OF) +#error "INDEX_OF not defined" +#endif + +#if !defined(MODNN) +#error "MODNN not defined" +#endif + +#if !defined(FCR) +#error "FCR not defined" +#endif + +#if !defined(PRIM) +#error "PRIM not defined" +#endif + +#if !defined(NULL) +#define NULL ((void *)0) +#endif + +#undef MIN +#define MIN(a,b) ((a) < (b) ? (a) : (b)) +#undef A0 +#define A0 (NN) + +{ + int deg_lambda, el, deg_omega; + int i, j, r,k; + data_t u,q,tmp,num1,num2,den,discr_r; + data_t lambda[NROOTS+1], s[NROOTS]; /* Err+Eras Locator poly + * and syndrome poly */ + data_t b[NROOTS+1], t[NROOTS+1], omega[NROOTS+1]; + data_t root[NROOTS], reg[NROOTS+1], loc[NROOTS]; + int syn_error, count; + + /* form the syndromes; i.e., evaluate data(x) at roots of g(x) */ + for(i=0;i<NROOTS;i++) + s[i] = data[0]; + + for(j=1;j<NN-PAD;j++){ + for(i=0;i<NROOTS;i++){ + if(s[i] == 0){ + s[i] = data[j]; + } else { + s[i] = data[j] ^ ALPHA_TO[MODNN(INDEX_OF[s[i]] + (FCR+i)*PRIM)]; + } + } + } + + /* Convert syndromes to index form, checking for nonzero condition */ + syn_error = 0; + for(i=0;i<NROOTS;i++){ + syn_error |= s[i]; + s[i] = INDEX_OF[s[i]]; + } + + if (!syn_error) { + /* if syndrome is zero, data[] is a codeword and there are no + * errors to correct. So return data[] unmodified + */ + count = 0; + goto finish; + } + memset(&lambda[1],0,NROOTS*sizeof(lambda[0])); + lambda[0] = 1; + + if (no_eras > 0) { + /* Init lambda to be the erasure locator polynomial */ + lambda[1] = ALPHA_TO[MODNN(PRIM*(NN-1-eras_pos[0]))]; + for (i = 1; i < no_eras; i++) { + u = MODNN(PRIM*(NN-1-eras_pos[i])); + for (j = i+1; j > 0; j--) { + tmp = INDEX_OF[lambda[j - 1]]; + if(tmp != A0) + lambda[j] ^= ALPHA_TO[MODNN(u + tmp)]; + } + } + +#if DEBUG >= 1 + /* Test code that verifies the erasure locator polynomial just constructed + Needed only for decoder debugging. */ + + /* find roots of the erasure location polynomial */ + for(i=1;i<=no_eras;i++) + reg[i] = INDEX_OF[lambda[i]]; + + count = 0; + for (i = 1,k=IPRIM-1; i <= NN; i++,k = MODNN(k+IPRIM)) { + q = 1; + for (j = 1; j <= no_eras; j++) + if (reg[j] != A0) { + reg[j] = MODNN(reg[j] + j); + q ^= ALPHA_TO[reg[j]]; + } + if (q != 0) + continue; + /* store root and error location number indices */ + root[count] = i; + loc[count] = k; + count++; + } + if (count != no_eras) { + printf("count = %d no_eras = %d\n lambda(x) is WRONG\n",count,no_eras); + count = -1; + goto finish; + } +#if DEBUG >= 2 + printf("\n Erasure positions as determined by roots of Eras Loc Poly:\n"); + for (i = 0; i < count; i++) + printf("%d ", loc[i]); + printf("\n"); +#endif +#endif + } + for(i=0;i<NROOTS+1;i++) + b[i] = INDEX_OF[lambda[i]]; + + /* + * Begin Berlekamp-Massey algorithm to determine error+erasure + * locator polynomial + */ + r = no_eras; + el = no_eras; + while (++r <= NROOTS) { /* r is the step number */ + /* Compute discrepancy at the r-th step in poly-form */ + discr_r = 0; + for (i = 0; i < r; i++){ + if ((lambda[i] != 0) && (s[r-i-1] != A0)) { + discr_r ^= ALPHA_TO[MODNN(INDEX_OF[lambda[i]] + s[r-i-1])]; + } + } + discr_r = INDEX_OF[discr_r]; /* Index form */ + if (discr_r == A0) { + /* 2 lines below: B(x) <-- x*B(x) */ + memmove(&b[1],b,NROOTS*sizeof(b[0])); + b[0] = A0; + } else { + /* 7 lines below: T(x) <-- lambda(x) - discr_r*x*b(x) */ + t[0] = lambda[0]; + for (i = 0 ; i < NROOTS; i++) { + if(b[i] != A0) + t[i+1] = lambda[i+1] ^ ALPHA_TO[MODNN(discr_r + b[i])]; + else + t[i+1] = lambda[i+1]; + } + if (2 * el <= r + no_eras - 1) { + el = r + no_eras - el; + /* + * 2 lines below: B(x) <-- inv(discr_r) * + * lambda(x) + */ + for (i = 0; i <= NROOTS; i++) + b[i] = (lambda[i] == 0) ? A0 : MODNN(INDEX_OF[lambda[i]] - discr_r + NN); + } else { + /* 2 lines below: B(x) <-- x*B(x) */ + memmove(&b[1],b,NROOTS*sizeof(b[0])); + b[0] = A0; + } + memcpy(lambda,t,(NROOTS+1)*sizeof(t[0])); + } + } + + /* Convert lambda to index form and compute deg(lambda(x)) */ + deg_lambda = 0; + for(i=0;i<NROOTS+1;i++){ + lambda[i] = INDEX_OF[lambda[i]]; + if(lambda[i] != A0) + deg_lambda = i; + } + /* Find roots of the error+erasure locator polynomial by Chien search */ + memcpy(®[1],&lambda[1],NROOTS*sizeof(reg[0])); + count = 0; /* Number of roots of lambda(x) */ + for (i = 1,k=IPRIM-1; i <= NN; i++,k = MODNN(k+IPRIM)) { + q = 1; /* lambda[0] is always 0 */ + for (j = deg_lambda; j > 0; j--){ + if (reg[j] != A0) { + reg[j] = MODNN(reg[j] + j); + q ^= ALPHA_TO[reg[j]]; + } + } + if (q != 0) + continue; /* Not a root */ + /* store root (index-form) and error location number */ +#if DEBUG>=2 + printf("count %d root %d loc %d\n",count,i,k); +#endif + root[count] = i; + loc[count] = k; + /* If we've already found max possible roots, + * abort the search to save time + */ + if(++count == deg_lambda) + break; + } + if (deg_lambda != count) { + /* + * deg(lambda) unequal to number of roots => uncorrectable + * error detected + */ + count = -1; + goto finish; + } + /* + * Compute err+eras evaluator poly omega(x) = s(x)*lambda(x) (modulo + * x**NROOTS). in index form. Also find deg(omega). + */ + deg_omega = deg_lambda-1; + for (i = 0; i <= deg_omega;i++){ + tmp = 0; + for(j=i;j >= 0; j--){ + if ((s[i - j] != A0) && (lambda[j] != A0)) + tmp ^= ALPHA_TO[MODNN(s[i - j] + lambda[j])]; + } + omega[i] = INDEX_OF[tmp]; + } + + /* + * Compute error values in poly-form. num1 = omega(inv(X(l))), num2 = + * inv(X(l))**(FCR-1) and den = lambda_pr(inv(X(l))) all in poly-form + */ + for (j = count-1; j >=0; j--) { + num1 = 0; + for (i = deg_omega; i >= 0; i--) { + if (omega[i] != A0) + num1 ^= ALPHA_TO[MODNN(omega[i] + i * root[j])]; + } + num2 = ALPHA_TO[MODNN(root[j] * (FCR - 1) + NN)]; + den = 0; + + /* lambda[i+1] for i even is the formal derivative lambda_pr of lambda[i] */ + for (i = MIN(deg_lambda,NROOTS-1) & ~1; i >= 0; i -=2) { + if(lambda[i+1] != A0) + den ^= ALPHA_TO[MODNN(lambda[i+1] + i * root[j])]; + } +#if DEBUG >= 1 + if (den == 0) { + printf("\n ERROR: denominator = 0\n"); + count = -1; + goto finish; + } +#endif + /* Apply error to data */ + if (num1 != 0 && loc[j] >= PAD) { + data[loc[j]-PAD] ^= ALPHA_TO[MODNN(INDEX_OF[num1] + INDEX_OF[num2] + NN - INDEX_OF[den])]; + } + } + finish: + if(eras_pos != NULL){ + for(i=0;i<count;i++) + eras_pos[i] = loc[i]; + } + retval = count; +} diff --git a/lib/fec/decode_rs_char.c b/lib/fec/decode_rs_char.c new file mode 100644 index 0000000..7105233 --- /dev/null +++ b/lib/fec/decode_rs_char.c @@ -0,0 +1,22 @@ +/* General purpose Reed-Solomon decoder for 8-bit symbols or less + * Copyright 2003 Phil Karn, KA9Q + * May be used under the terms of the GNU Lesser General Public License (LGPL) + */ + +#ifdef DEBUG +#include <stdio.h> +#endif + +#include <string.h> + +#include "char.h" +#include "rs-common.h" + +int decode_rs_char(void *p, data_t *data, int *eras_pos, int no_eras){ + int retval; + struct rs *rs = (struct rs *)p; + +#include "decode_rs.h" + + return retval; +} diff --git a/lib/fec/encode_rs.h b/lib/fec/encode_rs.h new file mode 100644 index 0000000..2c157f9 --- /dev/null +++ b/lib/fec/encode_rs.h @@ -0,0 +1,58 @@ +/* The guts of the Reed-Solomon encoder, meant to be #included + * into a function body with the following typedefs, macros and variables supplied + * according to the code parameters: + + * data_t - a typedef for the data symbol + * data_t data[] - array of NN-NROOTS-PAD and type data_t to be encoded + * data_t parity[] - an array of NROOTS and type data_t to be written with parity symbols + * NROOTS - the number of roots in the RS code generator polynomial, + * which is the same as the number of parity symbols in a block. + Integer variable or literal. + * + * NN - the total number of symbols in a RS block. Integer variable or literal. + * PAD - the number of pad symbols in a block. Integer variable or literal. + * ALPHA_TO - The address of an array of NN elements to convert Galois field + * elements in index (log) form to polynomial form. Read only. + * INDEX_OF - The address of an array of NN elements to convert Galois field + * elements in polynomial form to index (log) form. Read only. + * MODNN - a function to reduce its argument modulo NN. May be inline or a macro. + * GENPOLY - an array of NROOTS+1 elements containing the generator polynomial in index form + + * The memset() and memmove() functions are used. The appropriate header + * file declaring these functions (usually <string.h>) must be included by the calling + * program. + + * Copyright 2004, Phil Karn, KA9Q + * May be used under the terms of the GNU Lesser General Public License (LGPL) + */ + + +#undef A0 +#define A0 (NN) /* Special reserved value encoding zero in index form */ + +{ + int i, j; + data_t feedback; + + memset(parity,0,NROOTS*sizeof(data_t)); + + for(i=0;i<NN-NROOTS-PAD;i++){ + feedback = INDEX_OF[data[i] ^ parity[0]]; + if(feedback != A0){ /* feedback term is non-zero */ +#ifdef UNNORMALIZED + /* This line is unnecessary when GENPOLY[NROOTS] is unity, as it must + * always be for the polynomials constructed by init_rs() + */ + feedback = MODNN(NN - GENPOLY[NROOTS] + feedback); +#endif + for(j=1;j<NROOTS;j++) + parity[j] ^= ALPHA_TO[MODNN(feedback + GENPOLY[NROOTS-j])]; + } + /* Shift */ + memmove(&parity[0],&parity[1],sizeof(data_t)*(NROOTS-1)); + if(feedback != A0) + parity[NROOTS-1] = ALPHA_TO[MODNN(feedback + GENPOLY[0])]; + else + parity[NROOTS-1] = 0; + } +} diff --git a/lib/fec/encode_rs_char.c b/lib/fec/encode_rs_char.c new file mode 100644 index 0000000..a9bf2b8 --- /dev/null +++ b/lib/fec/encode_rs_char.c @@ -0,0 +1,15 @@ +/* Reed-Solomon encoder + * Copyright 2002, Phil Karn, KA9Q + * May be used under the terms of the GNU Lesser General Public License (LGPL) + */ +#include <string.h> + +#include "char.h" +#include "rs-common.h" + +void encode_rs_char(void *p,data_t *data, data_t *parity){ + struct rs *rs = (struct rs *)p; + +#include "encode_rs.h" + +} diff --git a/lib/fec/fec.h b/lib/fec/fec.h new file mode 100644 index 0000000..0d1bae1 --- /dev/null +++ b/lib/fec/fec.h @@ -0,0 +1,30 @@ +/* Main header for reduced libfec. + * + * The FEC code in this folder is + * Copyright 2003 Phil Karn, KA9Q + * May be used under the terms of the GNU Lesser General Public License (LGPL) + */ + +#pragma once + +#include <stdlib.h> + +#include "char.h" +#include "rs-common.h" + +/* Initialize a Reed-Solomon codec + * symsize = symbol size, bits + * gfpoly = Field generator polynomial coefficients + * fcr = first root of RS code generator polynomial, index form + * prim = primitive element to generate polynomial roots + * nroots = RS code generator polynomial degree (number of roots) + * pad = padding bytes at front of shortened block + */ +void *init_rs_char(int symsize,int gfpoly,int fcr,int prim,int nroots,int pad); + +int decode_rs_char(void *p, data_t *data, int *eras_pos, int no_eras); + +void encode_rs_char(void *p,data_t *data, data_t *parity); + +void free_rs_char(void *p); + diff --git a/lib/fec/init_rs.h b/lib/fec/init_rs.h new file mode 100644 index 0000000..2b2ae98 --- /dev/null +++ b/lib/fec/init_rs.h @@ -0,0 +1,106 @@ +/* Common code for intializing a Reed-Solomon control block (char or int symbols) + * Copyright 2004 Phil Karn, KA9Q + * May be used under the terms of the GNU Lesser General Public License (LGPL) + */ +#undef NULL +#define NULL ((void *)0) + +{ + int i, j, sr,root,iprim; + + rs = NULL; + /* Check parameter ranges */ + if(symsize < 0 || symsize > 8*sizeof(data_t)){ + goto done; + } + + if(fcr < 0 || fcr >= (1<<symsize)) + goto done; + if(prim <= 0 || prim >= (1<<symsize)) + goto done; + if(nroots < 0 || nroots >= (1<<symsize)) + goto done; /* Can't have more roots than symbol values! */ + if(pad < 0 || pad >= ((1<<symsize) -1 - nroots)) + goto done; /* Too much padding */ + + rs = (struct rs *)calloc(1,sizeof(struct rs)); + if(rs == NULL) + goto done; + + rs->mm = symsize; + rs->nn = (1<<symsize)-1; + rs->pad = pad; + + rs->alpha_to = (data_t *)malloc(sizeof(data_t)*(rs->nn+1)); + if(rs->alpha_to == NULL){ + free(rs); + rs = NULL; + goto done; + } + rs->index_of = (data_t *)malloc(sizeof(data_t)*(rs->nn+1)); + if(rs->index_of == NULL){ + free(rs->alpha_to); + free(rs); + rs = NULL; + goto done; + } + + /* Generate Galois field lookup tables */ + rs->index_of[0] = A0; /* log(zero) = -inf */ + rs->alpha_to[A0] = 0; /* alpha**-inf = 0 */ + sr = 1; + for(i=0;i<rs->nn;i++){ + rs->index_of[sr] = i; + rs->alpha_to[i] = sr; + sr <<= 1; + if(sr & (1<<symsize)) + sr ^= gfpoly; + sr &= rs->nn; + } + if(sr != 1){ + /* field generator polynomial is not primitive! */ + free(rs->alpha_to); + free(rs->index_of); + free(rs); + rs = NULL; + goto done; + } + + /* Form RS code generator polynomial from its roots */ + rs->genpoly = (data_t *)malloc(sizeof(data_t)*(nroots+1)); + if(rs->genpoly == NULL){ + free(rs->alpha_to); + free(rs->index_of); + free(rs); + rs = NULL; + goto done; + } + rs->fcr = fcr; + rs->prim = prim; + rs->nroots = nroots; + + /* Find prim-th root of 1, used in decoding */ + for(iprim=1;(iprim % prim) != 0;iprim += rs->nn) + ; + rs->iprim = iprim / prim; + + rs->genpoly[0] = 1; + for (i = 0,root=fcr*prim; i < nroots; i++,root += prim) { + rs->genpoly[i+1] = 1; + + /* Multiply rs->genpoly[] by @**(root + x) */ + for (j = i; j > 0; j--){ + if (rs->genpoly[j] != 0) + rs->genpoly[j] = rs->genpoly[j-1] ^ rs->alpha_to[modnn(rs,rs->index_of[rs->genpoly[j]] + root)]; + else + rs->genpoly[j] = rs->genpoly[j-1]; + } + /* rs->genpoly[0] can never be zero */ + rs->genpoly[0] = rs->alpha_to[modnn(rs,rs->index_of[rs->genpoly[0]] + root)]; + } + /* convert rs->genpoly[] to index form for quicker encoding */ + for (i = 0; i <= nroots; i++) + rs->genpoly[i] = rs->index_of[rs->genpoly[i]]; + done:; + +} diff --git a/lib/fec/init_rs_char.c b/lib/fec/init_rs_char.c new file mode 100644 index 0000000..a51099a --- /dev/null +++ b/lib/fec/init_rs_char.c @@ -0,0 +1,35 @@ +/* Initialize a RS codec + * + * Copyright 2002 Phil Karn, KA9Q + * May be used under the terms of the GNU Lesser General Public License (LGPL) + */ +#include <stdlib.h> + +#include "char.h" +#include "rs-common.h" + +void free_rs_char(void *p){ + struct rs *rs = (struct rs *)p; + + free(rs->alpha_to); + free(rs->index_of); + free(rs->genpoly); + free(rs); +} + +/* Initialize a Reed-Solomon codec + * symsize = symbol size, bits + * gfpoly = Field generator polynomial coefficients + * fcr = first root of RS code generator polynomial, index form + * prim = primitive element to generate polynomial roots + * nroots = RS code generator polynomial degree (number of roots) + * pad = padding bytes at front of shortened block + */ +void *init_rs_char(int symsize,int gfpoly,int fcr,int prim, + int nroots,int pad){ + struct rs *rs; + +#include "init_rs.h" + + return rs; +} diff --git a/lib/fec/rs-common.h b/lib/fec/rs-common.h new file mode 100644 index 0000000..e64eb39 --- /dev/null +++ b/lib/fec/rs-common.h @@ -0,0 +1,26 @@ +/* Stuff common to all the general-purpose Reed-Solomon codecs + * Copyright 2004 Phil Karn, KA9Q + * May be used under the terms of the GNU Lesser General Public License (LGPL) + */ + +/* Reed-Solomon codec control block */ +struct rs { + int mm; /* Bits per symbol */ + int nn; /* Symbols per block (= (1<<mm)-1) */ + data_t *alpha_to; /* log lookup table */ + data_t *index_of; /* Antilog lookup table */ + data_t *genpoly; /* Generator polynomial */ + int nroots; /* Number of generator roots = number of parity symbols */ + int fcr; /* First consecutive root, index form */ + int prim; /* Primitive element, index form */ + int iprim; /* prim-th root of 1, index form */ + int pad; /* Padding bytes in shortened block */ +}; + +static inline int modnn(struct rs *rs,int x){ + while (x >= rs->nn) { + x -= rs->nn; + x = (x >> rs->mm) + (x & rs->nn); + } + return x; +} diff --git a/m4/ax_check_compile_flag.m4 b/m4/ax_check_compile_flag.m4 new file mode 100644 index 0000000..bd753b3 --- /dev/null +++ b/m4/ax_check_compile_flag.m4 @@ -0,0 +1,53 @@ +# =========================================================================== +# https://www.gnu.org/software/autoconf-archive/ax_check_compile_flag.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_CHECK_COMPILE_FLAG(FLAG, [ACTION-SUCCESS], [ACTION-FAILURE], [EXTRA-FLAGS], [INPUT]) +# +# DESCRIPTION +# +# Check whether the given FLAG works with the current language's compiler +# or gives an error. (Warnings, however, are ignored) +# +# ACTION-SUCCESS/ACTION-FAILURE are shell commands to execute on +# success/failure. +# +# If EXTRA-FLAGS is defined, it is added to the current language's default +# flags (e.g. CFLAGS) when the check is done. The check is thus made with +# the flags: "CFLAGS EXTRA-FLAGS FLAG". This can for example be used to +# force the compiler to issue an error when a bad flag is given. +# +# INPUT gives an alternative input source to AC_COMPILE_IFELSE. +# +# NOTE: Implementation based on AX_CFLAGS_GCC_OPTION. Please keep this +# macro in sync with AX_CHECK_{PREPROC,LINK}_FLAG. +# +# LICENSE +# +# Copyright (c) 2008 Guido U. Draheim <guidod@gmx.de> +# Copyright (c) 2011 Maarten Bosmans <mkbosmans@gmail.com> +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice +# and this notice are preserved. This file is offered as-is, without any +# warranty. + +#serial 6 + +AC_DEFUN([AX_CHECK_COMPILE_FLAG], +[AC_PREREQ(2.64)dnl for _AC_LANG_PREFIX and AS_VAR_IF +AS_VAR_PUSHDEF([CACHEVAR],[ax_cv_check_[]_AC_LANG_ABBREV[]flags_$4_$1])dnl +AC_CACHE_CHECK([whether _AC_LANG compiler accepts $1], CACHEVAR, [ + ax_check_save_flags=$[]_AC_LANG_PREFIX[]FLAGS + _AC_LANG_PREFIX[]FLAGS="$[]_AC_LANG_PREFIX[]FLAGS $4 $1" + AC_COMPILE_IFELSE([m4_default([$5],[AC_LANG_PROGRAM()])], + [AS_VAR_SET(CACHEVAR,[yes])], + [AS_VAR_SET(CACHEVAR,[no])]) + _AC_LANG_PREFIX[]FLAGS=$ax_check_save_flags]) +AS_VAR_IF(CACHEVAR,yes, + [m4_default([$2], :)], + [m4_default([$3], :)]) +AS_VAR_POPDEF([CACHEVAR])dnl +])dnl AX_CHECK_COMPILE_FLAGS diff --git a/m4/ax_cxx_compile_stdcxx.m4 b/m4/ax_cxx_compile_stdcxx.m4 new file mode 100644 index 0000000..43087b2 --- /dev/null +++ b/m4/ax_cxx_compile_stdcxx.m4 @@ -0,0 +1,951 @@ +# =========================================================================== +# https://www.gnu.org/software/autoconf-archive/ax_cxx_compile_stdcxx.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_CXX_COMPILE_STDCXX(VERSION, [ext|noext], [mandatory|optional]) +# +# DESCRIPTION +# +# Check for baseline language coverage in the compiler for the specified +# version of the C++ standard. If necessary, add switches to CXX and +# CXXCPP to enable support. VERSION may be '11' (for the C++11 standard) +# or '14' (for the C++14 standard). +# +# The second argument, if specified, indicates whether you insist on an +# extended mode (e.g. -std=gnu++11) or a strict conformance mode (e.g. +# -std=c++11). If neither is specified, you get whatever works, with +# preference for an extended mode. +# +# The third argument, if specified 'mandatory' or if left unspecified, +# indicates that baseline support for the specified C++ standard is +# required and that the macro should error out if no mode with that +# support is found. If specified 'optional', then configuration proceeds +# regardless, after defining HAVE_CXX${VERSION} if and only if a +# supporting mode is found. +# +# LICENSE +# +# Copyright (c) 2008 Benjamin Kosnik <bkoz@redhat.com> +# Copyright (c) 2012 Zack Weinberg <zackw@panix.com> +# Copyright (c) 2013 Roy Stogner <roystgnr@ices.utexas.edu> +# Copyright (c) 2014, 2015 Google Inc.; contributed by Alexey Sokolov <sokolov@google.com> +# Copyright (c) 2015 Paul Norman <penorman@mac.com> +# Copyright (c) 2015 Moritz Klammler <moritz@klammler.eu> +# Copyright (c) 2016, 2018 Krzesimir Nowak <qdlacz@gmail.com> +# Copyright (c) 2019 Enji Cooper <yaneurabeya@gmail.com> +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice +# and this notice are preserved. This file is offered as-is, without any +# warranty. + +#serial 11 + +dnl This macro is based on the code from the AX_CXX_COMPILE_STDCXX_11 macro +dnl (serial version number 13). + +AC_DEFUN([AX_CXX_COMPILE_STDCXX], [dnl + m4_if([$1], [11], [ax_cxx_compile_alternatives="11 0x"], + [$1], [14], [ax_cxx_compile_alternatives="14 1y"], + [$1], [17], [ax_cxx_compile_alternatives="17 1z"], + [m4_fatal([invalid first argument `$1' to AX_CXX_COMPILE_STDCXX])])dnl + m4_if([$2], [], [], + [$2], [ext], [], + [$2], [noext], [], + [m4_fatal([invalid second argument `$2' to AX_CXX_COMPILE_STDCXX])])dnl + m4_if([$3], [], [ax_cxx_compile_cxx$1_required=true], + [$3], [mandatory], [ax_cxx_compile_cxx$1_required=true], + [$3], [optional], [ax_cxx_compile_cxx$1_required=false], + [m4_fatal([invalid third argument `$3' to AX_CXX_COMPILE_STDCXX])]) + AC_LANG_PUSH([C++])dnl + ac_success=no + + m4_if([$2], [noext], [], [dnl + if test x$ac_success = xno; then + for alternative in ${ax_cxx_compile_alternatives}; do + switch="-std=gnu++${alternative}" + cachevar=AS_TR_SH([ax_cv_cxx_compile_cxx$1_$switch]) + AC_CACHE_CHECK(whether $CXX supports C++$1 features with $switch, + $cachevar, + [ac_save_CXX="$CXX" + CXX="$CXX $switch" + AC_COMPILE_IFELSE([AC_LANG_SOURCE([_AX_CXX_COMPILE_STDCXX_testbody_$1])], + [eval $cachevar=yes], + [eval $cachevar=no]) + CXX="$ac_save_CXX"]) + if eval test x\$$cachevar = xyes; then + CXX="$CXX $switch" + if test -n "$CXXCPP" ; then + CXXCPP="$CXXCPP $switch" + fi + ac_success=yes + break + fi + done + fi]) + + m4_if([$2], [ext], [], [dnl + if test x$ac_success = xno; then + dnl HP's aCC needs +std=c++11 according to: + dnl http://h21007.www2.hp.com/portal/download/files/unprot/aCxx/PDF_Release_Notes/769149-001.pdf + dnl Cray's crayCC needs "-h std=c++11" + for alternative in ${ax_cxx_compile_alternatives}; do + for switch in -std=c++${alternative} +std=c++${alternative} "-h std=c++${alternative}"; do + cachevar=AS_TR_SH([ax_cv_cxx_compile_cxx$1_$switch]) + AC_CACHE_CHECK(whether $CXX supports C++$1 features with $switch, + $cachevar, + [ac_save_CXX="$CXX" + CXX="$CXX $switch" + AC_COMPILE_IFELSE([AC_LANG_SOURCE([_AX_CXX_COMPILE_STDCXX_testbody_$1])], + [eval $cachevar=yes], + [eval $cachevar=no]) + CXX="$ac_save_CXX"]) + if eval test x\$$cachevar = xyes; then + CXX="$CXX $switch" + if test -n "$CXXCPP" ; then + CXXCPP="$CXXCPP $switch" + fi + ac_success=yes + break + fi + done + if test x$ac_success = xyes; then + break + fi + done + fi]) + AC_LANG_POP([C++]) + if test x$ax_cxx_compile_cxx$1_required = xtrue; then + if test x$ac_success = xno; then + AC_MSG_ERROR([*** A compiler with support for C++$1 language features is required.]) + fi + fi + if test x$ac_success = xno; then + HAVE_CXX$1=0 + AC_MSG_NOTICE([No compiler with C++$1 support was found]) + else + HAVE_CXX$1=1 + AC_DEFINE(HAVE_CXX$1,1, + [define if the compiler supports basic C++$1 syntax]) + fi + AC_SUBST(HAVE_CXX$1) +]) + + +dnl Test body for checking C++11 support + +m4_define([_AX_CXX_COMPILE_STDCXX_testbody_11], + _AX_CXX_COMPILE_STDCXX_testbody_new_in_11 +) + + +dnl Test body for checking C++14 support + +m4_define([_AX_CXX_COMPILE_STDCXX_testbody_14], + _AX_CXX_COMPILE_STDCXX_testbody_new_in_11 + _AX_CXX_COMPILE_STDCXX_testbody_new_in_14 +) + +m4_define([_AX_CXX_COMPILE_STDCXX_testbody_17], + _AX_CXX_COMPILE_STDCXX_testbody_new_in_11 + _AX_CXX_COMPILE_STDCXX_testbody_new_in_14 + _AX_CXX_COMPILE_STDCXX_testbody_new_in_17 +) + +dnl Tests for new features in C++11 + +m4_define([_AX_CXX_COMPILE_STDCXX_testbody_new_in_11], [[ + +// If the compiler admits that it is not ready for C++11, why torture it? +// Hopefully, this will speed up the test. + +#ifndef __cplusplus + +#error "This is not a C++ compiler" + +#elif __cplusplus < 201103L + +#error "This is not a C++11 compiler" + +#else + +namespace cxx11 +{ + + namespace test_static_assert + { + + template <typename T> + struct check + { + static_assert(sizeof(int) <= sizeof(T), "not big enough"); + }; + + } + + namespace test_final_override + { + + struct Base + { + virtual ~Base() {} + virtual void f() {} + }; + + struct Derived : public Base + { + virtual ~Derived() override {} + virtual void f() override {} + }; + + } + + namespace test_double_right_angle_brackets + { + + template < typename T > + struct check {}; + + typedef check<void> single_type; + typedef check<check<void>> double_type; + typedef check<check<check<void>>> triple_type; + typedef check<check<check<check<void>>>> quadruple_type; + + } + + namespace test_decltype + { + + int + f() + { + int a = 1; + decltype(a) b = 2; + return a + b; + } + + } + + namespace test_type_deduction + { + + template < typename T1, typename T2 > + struct is_same + { + static const bool value = false; + }; + + template < typename T > + struct is_same<T, T> + { + static const bool value = true; + }; + + template < typename T1, typename T2 > + auto + add(T1 a1, T2 a2) -> decltype(a1 + a2) + { + return a1 + a2; + } + + int + test(const int c, volatile int v) + { + static_assert(is_same<int, decltype(0)>::value == true, ""); + static_assert(is_same<int, decltype(c)>::value == false, ""); + static_assert(is_same<int, decltype(v)>::value == false, ""); + auto ac = c; + auto av = v; + auto sumi = ac + av + 'x'; + auto sumf = ac + av + 1.0; + static_assert(is_same<int, decltype(ac)>::value == true, ""); + static_assert(is_same<int, decltype(av)>::value == true, ""); + static_assert(is_same<int, decltype(sumi)>::value == true, ""); + static_assert(is_same<int, decltype(sumf)>::value == false, ""); + static_assert(is_same<int, decltype(add(c, v))>::value == true, ""); + return (sumf > 0.0) ? sumi : add(c, v); + } + + } + + namespace test_noexcept + { + + int f() { return 0; } + int g() noexcept { return 0; } + + static_assert(noexcept(f()) == false, ""); + static_assert(noexcept(g()) == true, ""); + + } + + namespace test_constexpr + { + + template < typename CharT > + unsigned long constexpr + strlen_c_r(const CharT *const s, const unsigned long acc) noexcept + { + return *s ? strlen_c_r(s + 1, acc + 1) : acc; + } + + template < typename CharT > + unsigned long constexpr + strlen_c(const CharT *const s) noexcept + { + return strlen_c_r(s, 0UL); + } + + static_assert(strlen_c("") == 0UL, ""); + static_assert(strlen_c("1") == 1UL, ""); + static_assert(strlen_c("example") == 7UL, ""); + static_assert(strlen_c("another\0example") == 7UL, ""); + + } + + namespace test_rvalue_references + { + + template < int N > + struct answer + { + static constexpr int value = N; + }; + + answer<1> f(int&) { return answer<1>(); } + answer<2> f(const int&) { return answer<2>(); } + answer<3> f(int&&) { return answer<3>(); } + + void + test() + { + int i = 0; + const int c = 0; + static_assert(decltype(f(i))::value == 1, ""); + static_assert(decltype(f(c))::value == 2, ""); + static_assert(decltype(f(0))::value == 3, ""); + } + + } + + namespace test_uniform_initialization + { + + struct test + { + static const int zero {}; + static const int one {1}; + }; + + static_assert(test::zero == 0, ""); + static_assert(test::one == 1, ""); + + } + + namespace test_lambdas + { + + void + test1() + { + auto lambda1 = [](){}; + auto lambda2 = lambda1; + lambda1(); + lambda2(); + } + + int + test2() + { + auto a = [](int i, int j){ return i + j; }(1, 2); + auto b = []() -> int { return '0'; }(); + auto c = [=](){ return a + b; }(); + auto d = [&](){ return c; }(); + auto e = [a, &b](int x) mutable { + const auto identity = [](int y){ return y; }; + for (auto i = 0; i < a; ++i) + a += b--; + return x + identity(a + b); + }(0); + return a + b + c + d + e; + } + + int + test3() + { + const auto nullary = [](){ return 0; }; + const auto unary = [](int x){ return x; }; + using nullary_t = decltype(nullary); + using unary_t = decltype(unary); + const auto higher1st = [](nullary_t f){ return f(); }; + const auto higher2nd = [unary](nullary_t f1){ + return [unary, f1](unary_t f2){ return f2(unary(f1())); }; + }; + return higher1st(nullary) + higher2nd(nullary)(unary); + } + + } + + namespace test_variadic_templates + { + + template <int...> + struct sum; + + template <int N0, int... N1toN> + struct sum<N0, N1toN...> + { + static constexpr auto value = N0 + sum<N1toN...>::value; + }; + + template <> + struct sum<> + { + static constexpr auto value = 0; + }; + + static_assert(sum<>::value == 0, ""); + static_assert(sum<1>::value == 1, ""); + static_assert(sum<23>::value == 23, ""); + static_assert(sum<1, 2>::value == 3, ""); + static_assert(sum<5, 5, 11>::value == 21, ""); + static_assert(sum<2, 3, 5, 7, 11, 13>::value == 41, ""); + + } + + // http://stackoverflow.com/questions/13728184/template-aliases-and-sfinae + // Clang 3.1 fails with headers of libstd++ 4.8.3 when using std::function + // because of this. + namespace test_template_alias_sfinae + { + + struct foo {}; + + template<typename T> + using member = typename T::member_type; + + template<typename T> + void func(...) {} + + template<typename T> + void func(member<T>*) {} + + void test(); + + void test() { func<foo>(0); } + + } + +} // namespace cxx11 + +#endif // __cplusplus >= 201103L + +]]) + + +dnl Tests for new features in C++14 + +m4_define([_AX_CXX_COMPILE_STDCXX_testbody_new_in_14], [[ + +// If the compiler admits that it is not ready for C++14, why torture it? +// Hopefully, this will speed up the test. + +#ifndef __cplusplus + +#error "This is not a C++ compiler" + +#elif __cplusplus < 201402L + +#error "This is not a C++14 compiler" + +#else + +namespace cxx14 +{ + + namespace test_polymorphic_lambdas + { + + int + test() + { + const auto lambda = [](auto&&... args){ + const auto istiny = [](auto x){ + return (sizeof(x) == 1UL) ? 1 : 0; + }; + const int aretiny[] = { istiny(args)... }; + return aretiny[0]; + }; + return lambda(1, 1L, 1.0f, '1'); + } + + } + + namespace test_binary_literals + { + + constexpr auto ivii = 0b0000000000101010; + static_assert(ivii == 42, "wrong value"); + + } + + namespace test_generalized_constexpr + { + + template < typename CharT > + constexpr unsigned long + strlen_c(const CharT *const s) noexcept + { + auto length = 0UL; + for (auto p = s; *p; ++p) + ++length; + return length; + } + + static_assert(strlen_c("") == 0UL, ""); + static_assert(strlen_c("x") == 1UL, ""); + static_assert(strlen_c("test") == 4UL, ""); + static_assert(strlen_c("another\0test") == 7UL, ""); + + } + + namespace test_lambda_init_capture + { + + int + test() + { + auto x = 0; + const auto lambda1 = [a = x](int b){ return a + b; }; + const auto lambda2 = [a = lambda1(x)](){ return a; }; + return lambda2(); + } + + } + + namespace test_digit_separators + { + + constexpr auto ten_million = 100'000'000; + static_assert(ten_million == 100000000, ""); + + } + + namespace test_return_type_deduction + { + + auto f(int& x) { return x; } + decltype(auto) g(int& x) { return x; } + + template < typename T1, typename T2 > + struct is_same + { + static constexpr auto value = false; + }; + + template < typename T > + struct is_same<T, T> + { + static constexpr auto value = true; + }; + + int + test() + { + auto x = 0; + static_assert(is_same<int, decltype(f(x))>::value, ""); + static_assert(is_same<int&, decltype(g(x))>::value, ""); + return x; + } + + } + +} // namespace cxx14 + +#endif // __cplusplus >= 201402L + +]]) + + +dnl Tests for new features in C++17 + +m4_define([_AX_CXX_COMPILE_STDCXX_testbody_new_in_17], [[ + +// If the compiler admits that it is not ready for C++17, why torture it? +// Hopefully, this will speed up the test. + +#ifndef __cplusplus + +#error "This is not a C++ compiler" + +#elif __cplusplus < 201703L + +#error "This is not a C++17 compiler" + +#else + +#include <initializer_list> +#include <utility> +#include <type_traits> + +namespace cxx17 +{ + + namespace test_constexpr_lambdas + { + + constexpr int foo = [](){return 42;}(); + + } + + namespace test::nested_namespace::definitions + { + + } + + namespace test_fold_expression + { + + template<typename... Args> + int multiply(Args... args) + { + return (args * ... * 1); + } + + template<typename... Args> + bool all(Args... args) + { + return (args && ...); + } + + } + + namespace test_extended_static_assert + { + + static_assert (true); + + } + + namespace test_auto_brace_init_list + { + + auto foo = {5}; + auto bar {5}; + + static_assert(std::is_same<std::initializer_list<int>, decltype(foo)>::value); + static_assert(std::is_same<int, decltype(bar)>::value); + } + + namespace test_typename_in_template_template_parameter + { + + template<template<typename> typename X> struct D; + + } + + namespace test_fallthrough_nodiscard_maybe_unused_attributes + { + + int f1() + { + return 42; + } + + [[nodiscard]] int f2() + { + [[maybe_unused]] auto unused = f1(); + + switch (f1()) + { + case 17: + f1(); + [[fallthrough]]; + case 42: + f1(); + } + return f1(); + } + + } + + namespace test_extended_aggregate_initialization + { + + struct base1 + { + int b1, b2 = 42; + }; + + struct base2 + { + base2() { + b3 = 42; + } + int b3; + }; + + struct derived : base1, base2 + { + int d; + }; + + derived d1 {{1, 2}, {}, 4}; // full initialization + derived d2 {{}, {}, 4}; // value-initialized bases + + } + + namespace test_general_range_based_for_loop + { + + struct iter + { + int i; + + int& operator* () + { + return i; + } + + const int& operator* () const + { + return i; + } + + iter& operator++() + { + ++i; + return *this; + } + }; + + struct sentinel + { + int i; + }; + + bool operator== (const iter& i, const sentinel& s) + { + return i.i == s.i; + } + + bool operator!= (const iter& i, const sentinel& s) + { + return !(i == s); + } + + struct range + { + iter begin() const + { + return {0}; + } + + sentinel end() const + { + return {5}; + } + }; + + void f() + { + range r {}; + + for (auto i : r) + { + [[maybe_unused]] auto v = i; + } + } + + } + + namespace test_lambda_capture_asterisk_this_by_value + { + + struct t + { + int i; + int foo() + { + return [*this]() + { + return i; + }(); + } + }; + + } + + namespace test_enum_class_construction + { + + enum class byte : unsigned char + {}; + + byte foo {42}; + + } + + namespace test_constexpr_if + { + + template <bool cond> + int f () + { + if constexpr(cond) + { + return 13; + } + else + { + return 42; + } + } + + } + + namespace test_selection_statement_with_initializer + { + + int f() + { + return 13; + } + + int f2() + { + if (auto i = f(); i > 0) + { + return 3; + } + + switch (auto i = f(); i + 4) + { + case 17: + return 2; + + default: + return 1; + } + } + + } + + namespace test_template_argument_deduction_for_class_templates + { + + template <typename T1, typename T2> + struct pair + { + pair (T1 p1, T2 p2) + : m1 {p1}, + m2 {p2} + {} + + T1 m1; + T2 m2; + }; + + void f() + { + [[maybe_unused]] auto p = pair{13, 42u}; + } + + } + + namespace test_non_type_auto_template_parameters + { + + template <auto n> + struct B + {}; + + B<5> b1; + B<'a'> b2; + + } + + namespace test_structured_bindings + { + + int arr[2] = { 1, 2 }; + std::pair<int, int> pr = { 1, 2 }; + + auto f1() -> int(&)[2] + { + return arr; + } + + auto f2() -> std::pair<int, int>& + { + return pr; + } + + struct S + { + int x1 : 2; + volatile double y1; + }; + + S f3() + { + return {}; + } + + auto [ x1, y1 ] = f1(); + auto& [ xr1, yr1 ] = f1(); + auto [ x2, y2 ] = f2(); + auto& [ xr2, yr2 ] = f2(); + const auto [ x3, y3 ] = f3(); + + } + + namespace test_exception_spec_type_system + { + + struct Good {}; + struct Bad {}; + + void g1() noexcept; + void g2(); + + template<typename T> + Bad + f(T*, T*); + + template<typename T1, typename T2> + Good + f(T1*, T2*); + + static_assert (std::is_same_v<Good, decltype(f(g1, g2))>); + + } + + namespace test_inline_variables + { + + template<class T> void f(T) + {} + + template<class T> inline T g(T) + { + return T{}; + } + + template<> inline void f<>(int) + {} + + template<> int g<>(int) + { + return 5; + } + + } + +} // namespace cxx17 + +#endif // __cplusplus < 201703L + +]]) diff --git a/m4/ax_pthread.m4 b/m4/ax_pthread.m4 new file mode 100644 index 0000000..1598d07 --- /dev/null +++ b/m4/ax_pthread.m4 @@ -0,0 +1,507 @@ +# =========================================================================== +# https://www.gnu.org/software/autoconf-archive/ax_pthread.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_PTHREAD([ACTION-IF-FOUND[, ACTION-IF-NOT-FOUND]]) +# +# DESCRIPTION +# +# This macro figures out how to build C programs using POSIX threads. It +# sets the PTHREAD_LIBS output variable to the threads library and linker +# flags, and the PTHREAD_CFLAGS output variable to any special C compiler +# flags that are needed. (The user can also force certain compiler +# flags/libs to be tested by setting these environment variables.) +# +# Also sets PTHREAD_CC to any special C compiler that is needed for +# multi-threaded programs (defaults to the value of CC otherwise). (This +# is necessary on AIX to use the special cc_r compiler alias.) +# +# NOTE: You are assumed to not only compile your program with these flags, +# but also to link with them as well. For example, you might link with +# $PTHREAD_CC $CFLAGS $PTHREAD_CFLAGS $LDFLAGS ... $PTHREAD_LIBS $LIBS +# +# If you are only building threaded programs, you may wish to use these +# variables in your default LIBS, CFLAGS, and CC: +# +# LIBS="$PTHREAD_LIBS $LIBS" +# CFLAGS="$CFLAGS $PTHREAD_CFLAGS" +# CC="$PTHREAD_CC" +# +# In addition, if the PTHREAD_CREATE_JOINABLE thread-attribute constant +# has a nonstandard name, this macro defines PTHREAD_CREATE_JOINABLE to +# that name (e.g. PTHREAD_CREATE_UNDETACHED on AIX). +# +# Also HAVE_PTHREAD_PRIO_INHERIT is defined if pthread is found and the +# PTHREAD_PRIO_INHERIT symbol is defined when compiling with +# PTHREAD_CFLAGS. +# +# ACTION-IF-FOUND is a list of shell commands to run if a threads library +# is found, and ACTION-IF-NOT-FOUND is a list of commands to run it if it +# is not found. If ACTION-IF-FOUND is not specified, the default action +# will define HAVE_PTHREAD. +# +# Please let the authors know if this macro fails on any platform, or if +# you have any other suggestions or comments. This macro was based on work +# by SGJ on autoconf scripts for FFTW (http://www.fftw.org/) (with help +# from M. Frigo), as well as ac_pthread and hb_pthread macros posted by +# Alejandro Forero Cuervo to the autoconf macro repository. We are also +# grateful for the helpful feedback of numerous users. +# +# Updated for Autoconf 2.68 by Daniel Richard G. +# +# LICENSE +# +# Copyright (c) 2008 Steven G. Johnson <stevenj@alum.mit.edu> +# Copyright (c) 2011 Daniel Richard G. <skunk@iSKUNK.ORG> +# Copyright (c) 2019 Marc Stevens <marc.stevens@cwi.nl> +# +# This program is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or (at your +# option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see <https://www.gnu.org/licenses/>. +# +# As a special exception, the respective Autoconf Macro's copyright owner +# gives unlimited permission to copy, distribute and modify the configure +# scripts that are the output of Autoconf when processing the Macro. You +# need not follow the terms of the GNU General Public License when using +# or distributing such scripts, even though portions of the text of the +# Macro appear in them. The GNU General Public License (GPL) does govern +# all other use of the material that constitutes the Autoconf Macro. +# +# This special exception to the GPL applies to versions of the Autoconf +# Macro released by the Autoconf Archive. When you make and distribute a +# modified version of the Autoconf Macro, you may extend this special +# exception to the GPL to apply to your modified version as well. + +#serial 27 + +AU_ALIAS([ACX_PTHREAD], [AX_PTHREAD]) +AC_DEFUN([AX_PTHREAD], [ +AC_REQUIRE([AC_CANONICAL_HOST]) +AC_REQUIRE([AC_PROG_CC]) +AC_REQUIRE([AC_PROG_SED]) +AC_LANG_PUSH([C]) +ax_pthread_ok=no + +# We used to check for pthread.h first, but this fails if pthread.h +# requires special compiler flags (e.g. on Tru64 or Sequent). +# It gets checked for in the link test anyway. + +# First of all, check if the user has set any of the PTHREAD_LIBS, +# etcetera environment variables, and if threads linking works using +# them: +if test "x$PTHREAD_CFLAGS$PTHREAD_LIBS" != "x"; then + ax_pthread_save_CC="$CC" + ax_pthread_save_CFLAGS="$CFLAGS" + ax_pthread_save_LIBS="$LIBS" + AS_IF([test "x$PTHREAD_CC" != "x"], [CC="$PTHREAD_CC"]) + CFLAGS="$CFLAGS $PTHREAD_CFLAGS" + LIBS="$PTHREAD_LIBS $LIBS" + AC_MSG_CHECKING([for pthread_join using $CC $PTHREAD_CFLAGS $PTHREAD_LIBS]) + AC_LINK_IFELSE([AC_LANG_CALL([], [pthread_join])], [ax_pthread_ok=yes]) + AC_MSG_RESULT([$ax_pthread_ok]) + if test "x$ax_pthread_ok" = "xno"; then + PTHREAD_LIBS="" + PTHREAD_CFLAGS="" + fi + CC="$ax_pthread_save_CC" + CFLAGS="$ax_pthread_save_CFLAGS" + LIBS="$ax_pthread_save_LIBS" +fi + +# We must check for the threads library under a number of different +# names; the ordering is very important because some systems +# (e.g. DEC) have both -lpthread and -lpthreads, where one of the +# libraries is broken (non-POSIX). + +# Create a list of thread flags to try. Items with a "," contain both +# C compiler flags (before ",") and linker flags (after ","). Other items +# starting with a "-" are C compiler flags, and remaining items are +# library names, except for "none" which indicates that we try without +# any flags at all, and "pthread-config" which is a program returning +# the flags for the Pth emulation library. + +ax_pthread_flags="pthreads none -Kthread -pthread -pthreads -mthreads pthread --thread-safe -mt pthread-config" + +# The ordering *is* (sometimes) important. Some notes on the +# individual items follow: + +# pthreads: AIX (must check this before -lpthread) +# none: in case threads are in libc; should be tried before -Kthread and +# other compiler flags to prevent continual compiler warnings +# -Kthread: Sequent (threads in libc, but -Kthread needed for pthread.h) +# -pthread: Linux/gcc (kernel threads), BSD/gcc (userland threads), Tru64 +# (Note: HP C rejects this with "bad form for `-t' option") +# -pthreads: Solaris/gcc (Note: HP C also rejects) +# -mt: Sun Workshop C (may only link SunOS threads [-lthread], but it +# doesn't hurt to check since this sometimes defines pthreads and +# -D_REENTRANT too), HP C (must be checked before -lpthread, which +# is present but should not be used directly; and before -mthreads, +# because the compiler interprets this as "-mt" + "-hreads") +# -mthreads: Mingw32/gcc, Lynx/gcc +# pthread: Linux, etcetera +# --thread-safe: KAI C++ +# pthread-config: use pthread-config program (for GNU Pth library) + +case $host_os in + + freebsd*) + + # -kthread: FreeBSD kernel threads (preferred to -pthread since SMP-able) + # lthread: LinuxThreads port on FreeBSD (also preferred to -pthread) + + ax_pthread_flags="-kthread lthread $ax_pthread_flags" + ;; + + hpux*) + + # From the cc(1) man page: "[-mt] Sets various -D flags to enable + # multi-threading and also sets -lpthread." + + ax_pthread_flags="-mt -pthread pthread $ax_pthread_flags" + ;; + + openedition*) + + # IBM z/OS requires a feature-test macro to be defined in order to + # enable POSIX threads at all, so give the user a hint if this is + # not set. (We don't define these ourselves, as they can affect + # other portions of the system API in unpredictable ways.) + + AC_EGREP_CPP([AX_PTHREAD_ZOS_MISSING], + [ +# if !defined(_OPEN_THREADS) && !defined(_UNIX03_THREADS) + AX_PTHREAD_ZOS_MISSING +# endif + ], + [AC_MSG_WARN([IBM z/OS requires -D_OPEN_THREADS or -D_UNIX03_THREADS to enable pthreads support.])]) + ;; + + solaris*) + + # On Solaris (at least, for some versions), libc contains stubbed + # (non-functional) versions of the pthreads routines, so link-based + # tests will erroneously succeed. (N.B.: The stubs are missing + # pthread_cleanup_push, or rather a function called by this macro, + # so we could check for that, but who knows whether they'll stub + # that too in a future libc.) So we'll check first for the + # standard Solaris way of linking pthreads (-mt -lpthread). + + ax_pthread_flags="-mt,-lpthread pthread $ax_pthread_flags" + ;; +esac + +# Are we compiling with Clang? + +AC_CACHE_CHECK([whether $CC is Clang], + [ax_cv_PTHREAD_CLANG], + [ax_cv_PTHREAD_CLANG=no + # Note that Autoconf sets GCC=yes for Clang as well as GCC + if test "x$GCC" = "xyes"; then + AC_EGREP_CPP([AX_PTHREAD_CC_IS_CLANG], + [/* Note: Clang 2.7 lacks __clang_[a-z]+__ */ +# if defined(__clang__) && defined(__llvm__) + AX_PTHREAD_CC_IS_CLANG +# endif + ], + [ax_cv_PTHREAD_CLANG=yes]) + fi + ]) +ax_pthread_clang="$ax_cv_PTHREAD_CLANG" + + +# GCC generally uses -pthread, or -pthreads on some platforms (e.g. SPARC) + +# Note that for GCC and Clang -pthread generally implies -lpthread, +# except when -nostdlib is passed. +# This is problematic using libtool to build C++ shared libraries with pthread: +# [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25460 +# [2] https://bugzilla.redhat.com/show_bug.cgi?id=661333 +# [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=468555 +# To solve this, first try -pthread together with -lpthread for GCC + +AS_IF([test "x$GCC" = "xyes"], + [ax_pthread_flags="-pthread,-lpthread -pthread -pthreads $ax_pthread_flags"]) + +# Clang takes -pthread (never supported any other flag), but we'll try with -lpthread first + +AS_IF([test "x$ax_pthread_clang" = "xyes"], + [ax_pthread_flags="-pthread,-lpthread -pthread"]) + + +# The presence of a feature test macro requesting re-entrant function +# definitions is, on some systems, a strong hint that pthreads support is +# correctly enabled + +case $host_os in + darwin* | hpux* | linux* | osf* | solaris*) + ax_pthread_check_macro="_REENTRANT" + ;; + + aix*) + ax_pthread_check_macro="_THREAD_SAFE" + ;; + + *) + ax_pthread_check_macro="--" + ;; +esac +AS_IF([test "x$ax_pthread_check_macro" = "x--"], + [ax_pthread_check_cond=0], + [ax_pthread_check_cond="!defined($ax_pthread_check_macro)"]) + + +if test "x$ax_pthread_ok" = "xno"; then +for ax_pthread_try_flag in $ax_pthread_flags; do + + case $ax_pthread_try_flag in + none) + AC_MSG_CHECKING([whether pthreads work without any flags]) + ;; + + *,*) + PTHREAD_CFLAGS=`echo $ax_pthread_try_flag | sed "s/^\(.*\),\(.*\)$/\1/"` + PTHREAD_LIBS=`echo $ax_pthread_try_flag | sed "s/^\(.*\),\(.*\)$/\2/"` + AC_MSG_CHECKING([whether pthreads work with "$PTHREAD_CFLAGS" and "$PTHREAD_LIBS"]) + ;; + + -*) + AC_MSG_CHECKING([whether pthreads work with $ax_pthread_try_flag]) + PTHREAD_CFLAGS="$ax_pthread_try_flag" + ;; + + pthread-config) + AC_CHECK_PROG([ax_pthread_config], [pthread-config], [yes], [no]) + AS_IF([test "x$ax_pthread_config" = "xno"], [continue]) + PTHREAD_CFLAGS="`pthread-config --cflags`" + PTHREAD_LIBS="`pthread-config --ldflags` `pthread-config --libs`" + ;; + + *) + AC_MSG_CHECKING([for the pthreads library -l$ax_pthread_try_flag]) + PTHREAD_LIBS="-l$ax_pthread_try_flag" + ;; + esac + + ax_pthread_save_CFLAGS="$CFLAGS" + ax_pthread_save_LIBS="$LIBS" + CFLAGS="$CFLAGS $PTHREAD_CFLAGS" + LIBS="$PTHREAD_LIBS $LIBS" + + # Check for various functions. We must include pthread.h, + # since some functions may be macros. (On the Sequent, we + # need a special flag -Kthread to make this header compile.) + # We check for pthread_join because it is in -lpthread on IRIX + # while pthread_create is in libc. We check for pthread_attr_init + # due to DEC craziness with -lpthreads. We check for + # pthread_cleanup_push because it is one of the few pthread + # functions on Solaris that doesn't have a non-functional libc stub. + # We try pthread_create on general principles. + + AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <pthread.h> +# if $ax_pthread_check_cond +# error "$ax_pthread_check_macro must be defined" +# endif + static void *some_global = NULL; + static void routine(void *a) + { + /* To avoid any unused-parameter or + unused-but-set-parameter warning. */ + some_global = a; + } + static void *start_routine(void *a) { return a; }], + [pthread_t th; pthread_attr_t attr; + pthread_create(&th, 0, start_routine, 0); + pthread_join(th, 0); + pthread_attr_init(&attr); + pthread_cleanup_push(routine, 0); + pthread_cleanup_pop(0) /* ; */])], + [ax_pthread_ok=yes], + []) + + CFLAGS="$ax_pthread_save_CFLAGS" + LIBS="$ax_pthread_save_LIBS" + + AC_MSG_RESULT([$ax_pthread_ok]) + AS_IF([test "x$ax_pthread_ok" = "xyes"], [break]) + + PTHREAD_LIBS="" + PTHREAD_CFLAGS="" +done +fi + + +# Clang needs special handling, because older versions handle the -pthread +# option in a rather... idiosyncratic way + +if test "x$ax_pthread_clang" = "xyes"; then + + # Clang takes -pthread; it has never supported any other flag + + # (Note 1: This will need to be revisited if a system that Clang + # supports has POSIX threads in a separate library. This tends not + # to be the way of modern systems, but it's conceivable.) + + # (Note 2: On some systems, notably Darwin, -pthread is not needed + # to get POSIX threads support; the API is always present and + # active. We could reasonably leave PTHREAD_CFLAGS empty. But + # -pthread does define _REENTRANT, and while the Darwin headers + # ignore this macro, third-party headers might not.) + + # However, older versions of Clang make a point of warning the user + # that, in an invocation where only linking and no compilation is + # taking place, the -pthread option has no effect ("argument unused + # during compilation"). They expect -pthread to be passed in only + # when source code is being compiled. + # + # Problem is, this is at odds with the way Automake and most other + # C build frameworks function, which is that the same flags used in + # compilation (CFLAGS) are also used in linking. Many systems + # supported by AX_PTHREAD require exactly this for POSIX threads + # support, and in fact it is often not straightforward to specify a + # flag that is used only in the compilation phase and not in + # linking. Such a scenario is extremely rare in practice. + # + # Even though use of the -pthread flag in linking would only print + # a warning, this can be a nuisance for well-run software projects + # that build with -Werror. So if the active version of Clang has + # this misfeature, we search for an option to squash it. + + AC_CACHE_CHECK([whether Clang needs flag to prevent "argument unused" warning when linking with -pthread], + [ax_cv_PTHREAD_CLANG_NO_WARN_FLAG], + [ax_cv_PTHREAD_CLANG_NO_WARN_FLAG=unknown + # Create an alternate version of $ac_link that compiles and + # links in two steps (.c -> .o, .o -> exe) instead of one + # (.c -> exe), because the warning occurs only in the second + # step + ax_pthread_save_ac_link="$ac_link" + ax_pthread_sed='s/conftest\.\$ac_ext/conftest.$ac_objext/g' + ax_pthread_link_step=`$as_echo "$ac_link" | sed "$ax_pthread_sed"` + ax_pthread_2step_ac_link="($ac_compile) && (echo ==== >&5) && ($ax_pthread_link_step)" + ax_pthread_save_CFLAGS="$CFLAGS" + for ax_pthread_try in '' -Qunused-arguments -Wno-unused-command-line-argument unknown; do + AS_IF([test "x$ax_pthread_try" = "xunknown"], [break]) + CFLAGS="-Werror -Wunknown-warning-option $ax_pthread_try -pthread $ax_pthread_save_CFLAGS" + ac_link="$ax_pthread_save_ac_link" + AC_LINK_IFELSE([AC_LANG_SOURCE([[int main(void){return 0;}]])], + [ac_link="$ax_pthread_2step_ac_link" + AC_LINK_IFELSE([AC_LANG_SOURCE([[int main(void){return 0;}]])], + [break]) + ]) + done + ac_link="$ax_pthread_save_ac_link" + CFLAGS="$ax_pthread_save_CFLAGS" + AS_IF([test "x$ax_pthread_try" = "x"], [ax_pthread_try=no]) + ax_cv_PTHREAD_CLANG_NO_WARN_FLAG="$ax_pthread_try" + ]) + + case "$ax_cv_PTHREAD_CLANG_NO_WARN_FLAG" in + no | unknown) ;; + *) PTHREAD_CFLAGS="$ax_cv_PTHREAD_CLANG_NO_WARN_FLAG $PTHREAD_CFLAGS" ;; + esac + +fi # $ax_pthread_clang = yes + + + +# Various other checks: +if test "x$ax_pthread_ok" = "xyes"; then + ax_pthread_save_CFLAGS="$CFLAGS" + ax_pthread_save_LIBS="$LIBS" + CFLAGS="$CFLAGS $PTHREAD_CFLAGS" + LIBS="$PTHREAD_LIBS $LIBS" + + # Detect AIX lossage: JOINABLE attribute is called UNDETACHED. + AC_CACHE_CHECK([for joinable pthread attribute], + [ax_cv_PTHREAD_JOINABLE_ATTR], + [ax_cv_PTHREAD_JOINABLE_ATTR=unknown + for ax_pthread_attr in PTHREAD_CREATE_JOINABLE PTHREAD_CREATE_UNDETACHED; do + AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <pthread.h>], + [int attr = $ax_pthread_attr; return attr /* ; */])], + [ax_cv_PTHREAD_JOINABLE_ATTR=$ax_pthread_attr; break], + []) + done + ]) + AS_IF([test "x$ax_cv_PTHREAD_JOINABLE_ATTR" != "xunknown" && \ + test "x$ax_cv_PTHREAD_JOINABLE_ATTR" != "xPTHREAD_CREATE_JOINABLE" && \ + test "x$ax_pthread_joinable_attr_defined" != "xyes"], + [AC_DEFINE_UNQUOTED([PTHREAD_CREATE_JOINABLE], + [$ax_cv_PTHREAD_JOINABLE_ATTR], + [Define to necessary symbol if this constant + uses a non-standard name on your system.]) + ax_pthread_joinable_attr_defined=yes + ]) + + AC_CACHE_CHECK([whether more special flags are required for pthreads], + [ax_cv_PTHREAD_SPECIAL_FLAGS], + [ax_cv_PTHREAD_SPECIAL_FLAGS=no + case $host_os in + solaris*) + ax_cv_PTHREAD_SPECIAL_FLAGS="-D_POSIX_PTHREAD_SEMANTICS" + ;; + esac + ]) + AS_IF([test "x$ax_cv_PTHREAD_SPECIAL_FLAGS" != "xno" && \ + test "x$ax_pthread_special_flags_added" != "xyes"], + [PTHREAD_CFLAGS="$ax_cv_PTHREAD_SPECIAL_FLAGS $PTHREAD_CFLAGS" + ax_pthread_special_flags_added=yes]) + + AC_CACHE_CHECK([for PTHREAD_PRIO_INHERIT], + [ax_cv_PTHREAD_PRIO_INHERIT], + [AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include <pthread.h>]], + [[int i = PTHREAD_PRIO_INHERIT; + return i;]])], + [ax_cv_PTHREAD_PRIO_INHERIT=yes], + [ax_cv_PTHREAD_PRIO_INHERIT=no]) + ]) + AS_IF([test "x$ax_cv_PTHREAD_PRIO_INHERIT" = "xyes" && \ + test "x$ax_pthread_prio_inherit_defined" != "xyes"], + [AC_DEFINE([HAVE_PTHREAD_PRIO_INHERIT], [1], [Have PTHREAD_PRIO_INHERIT.]) + ax_pthread_prio_inherit_defined=yes + ]) + + CFLAGS="$ax_pthread_save_CFLAGS" + LIBS="$ax_pthread_save_LIBS" + + # More AIX lossage: compile with *_r variant + if test "x$GCC" != "xyes"; then + case $host_os in + aix*) + AS_CASE(["x/$CC"], + [x*/c89|x*/c89_128|x*/c99|x*/c99_128|x*/cc|x*/cc128|x*/xlc|x*/xlc_v6|x*/xlc128|x*/xlc128_v6], + [#handle absolute path differently from PATH based program lookup + AS_CASE(["x$CC"], + [x/*], + [AS_IF([AS_EXECUTABLE_P([${CC}_r])],[PTHREAD_CC="${CC}_r"])], + [AC_CHECK_PROGS([PTHREAD_CC],[${CC}_r],[$CC])])]) + ;; + esac + fi +fi + +test -n "$PTHREAD_CC" || PTHREAD_CC="$CC" + +AC_SUBST([PTHREAD_LIBS]) +AC_SUBST([PTHREAD_CFLAGS]) +AC_SUBST([PTHREAD_CC]) + +# Finally, execute ACTION-IF-FOUND/ACTION-IF-NOT-FOUND: +if test "x$ax_pthread_ok" = "xyes"; then + ifelse([$1],,[AC_DEFINE([HAVE_PTHREAD],[1],[Define if you have POSIX threads libraries and header files.])],[$1]) + : +else + ax_pthread_ok=no + $2 +fi +AC_LANG_POP +])dnl AX_PTHREAD diff --git a/src/main.cpp b/src/main.cpp new file mode 100644 index 0000000..844908a --- /dev/null +++ b/src/main.cpp @@ -0,0 +1,31 @@ +/* + Copyright (C) 2020 + Matthias P. Braendli, matthias.braendli@mpb.li + + http://www.opendigitalradio.org + */ +/* + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + */ + +#include <iostream> +#include "Log.h" + +using namespace std; + +int main(int argc, char **argv) +{ + etiLog.level(info) << "Hello"; +} + |