[4.4.3] - 2022-12-23

Added

• None Nothing added

Changed

• None Nothing changed

Fixed

• Fix The changes for the last release caused one issue w/ unverifiable certificates, where the error isn't thrown when the connection is created, needed to add a connect() call at that time to catch the error; that was happening before, in a call I moved till later, which is why the error wasn't being caught on the creation where I was checking for it. Again, no coding chages for anyone, just made it throw the error where I was checjing for it.

Removed

• None Nothing removed

[4.4.2] - 2022-12-23

Added

• None Nothing added

Changed

• None Nothing changed

Fixed

• Fix I caused a regression with Viya 3.5 from having to rework authentication due to chages in Viya 4. This fixes that and requires no coding changes. This fixes issue #500

• Fix I fixed another issue with the same reworking as above that presented in another specific case. This fixes that and requires no coding changes. This fixes issue #503

Removed

• None Nothing removed

[4.4.1] - 2022-12-01

Added

• Enhanced I added a number of security type enhancements to the repo; none of which has to do with how the code works, and requires no code changes. This was all based upon OpenSSF best practices for Cybersecurity regarding open source repos. You can see the new badge on the home page: in the README.

Changed

• None Nothing changed

Fixed

• Fix I made a fix to the IOM access method based upon a vulnerability scan from the newly added SAST tool that was part of the Security based enhancements added to the repo. This requires no code changes.

• Fix I made a fix for the SAS_kernel, which depends upon SASPY to do the work interacting with SAS. There was a breaking change in the Kernel interface regarding prompting. I fixed it so it still works with previous versions (of course; no breaking changes for my customers!), as well as with the new api. This fix addresses SAS_kernel issue https://github.com/sassoftware/sas_kernel/issues/83

Removed

• None Nothing removed

[4.4.0] - 2022-11-14

Added

• Enhanced @rayewright added a half dozen more ML procs to the sasViyaML package for Viya.

Changed

• Enhanced Added documentation specifically identifying datatype conversions between SAS data Sets and Pandas dataframes; both directions. This was added in the AdvancedTopics section of the doc and the other couple parts of that section also about data movement were moved to all be contiguous and thus make all more clear. No code changes, just documentation.

• Enhanced Added documentation for the IOM access method regarding authentication. The Configurations section already specifies how to authenticate with different means, but now there's a part of the doc explicitly identifying which methods are supported and the one method that is not supported; SAS Token Authentication.

• Enhanced @andyjessen cleaned up some links in the doc that were still referring to master instead of main for the branch they linked to.

• Enhanced The HTTP access method, to Viya, requires a valid authentication token be passes to every request. This token is acquired when calling SASsession(). This token had been set to expire after 10 hours. Viya has been changed to have these tokens expire after 1 hour, so to keep this from causing problems for SASPy sessions, which can be interactive and last much longer than one hour, I've added support to reauthenticate and get a new authtoken prior to the current one expiring. This happens as long as the SASsession object is valid and connected. There are no coding changes required in user code. This should all just happen under the covers with no requirements on the user or program. I did add a method for explicitly refreshing the authtoken, but that should not be required at any time. But, it may be of use as a diagnostic in the field if there are ever issues reauthenticating. So, again, no code changes required.

Fixed

• None Nothing fixed

Removed

• None Nothing removed

[4.3.5] - 2022-10-17

Added

• None Nothing Added

Changed

• Tweak Had a PR with fixes to Doc; mostly typos and consistent use of terms. Also fixed a broken link.

Fixed

• Fix Fix for leaked resources. The pipes that are created between subprocess and Python were not being released prior to subtask termination. This results in a resource leak that is identified when running with settings that report these issues. I clean this up in both the IOM and STDIO access methods, they each had similar concerns. No programming changes required.

• Fix Fix for issue 487 where, in the df2sd() method, there was data that contained the data step termination string which resulted in the data step, which was retrieving the data and writing it to the SAS data step, terminating prior to processing all of the data and then SAS would take the rest of the stream as SAS code, which fails miserably and consumes memory with all of the errors going to the log. It also terminates the connection to the client. I've also added code in the java client to catch this kind of failure so it wouldn't hang, like it was. So this is addressed and fixed in this release, and requires no programming changes.

Removed

• None Nothing removed

[4.3.4] - 2022-10-05

Added

• None Nothing Added

Changed

• Tweak I'm moving to a newer dev environment; newer OS version, newer Python version, and all the newer modules, ... Using Sphinx to build the doc fails with this newer version, so I've had to tweak and reworks things to get the doc to build with this newer version of evrerything. So, there were a lot of minor adjustments pushed to accomodate this. It's all working and, of course, requires no programming changes due to any of this. A plus is that now the index, shown on the left side of the page, finally has links to the individual methods, in the API section of the doc! This is a welcome enhancement.

Fixed

• Fix Fix for issue 480 was in the last version. That issue was a discrepency with how Python on MacOS works compared to any other operating system. I've had to implement workarounds for Mac Python for a number of issues in the past. This release has a couple more workarouds for problems with the Mac version of Python too. These are in the IOM access method, whereas the last versions fix was in the STDIO access method. Again, no programming changes required.

• Fix All of the analytic methods correspond to some SAS Procedure. Each method is one analytic PROC. The code that generates the PROC syntax for all of tese methods is a bit convoluted. While fixing the method signatures for the doc building issues, last release, I noticed that if the USER librfer is assigned, then none of the analytic method work. I've worked through that code to resolve this, so in this release, that problem is fixed and they all work as they should have regardless of is USER is assigned or not.

Removed

• None Nothing removed

[4.3.3] - 2022-09-26

Added

• None Nothing Added

Changed

• Tweak The analytic methods all take a SASdata object, or the name of a SAS dataset (str). The method signatures used ['SASdata', str] to represent either/or, not a list. The current Spinx doc build no longer allows that and doesn't generate signatures, so I hade to replace the [] with (). No programming changes, just a tweak to get the doc to build right with the newer version of Spinx.

Fixed

• Fix Fix for issue 480. No programming changes required.

Removed

• None Nothing removed

[4.3.2] - 2022-08-12

Added

• Enhanced Added another configuration key in the SAS_config_options dictionary for providing an override for the ODS type. This was already available tochenge as an attribute on the SASSession object, but in a SAS_Kernel notebook, there was no way to do that. So I added style to the config so it could be set when using SAS_Kernel in Jupyter.

Changed

• Tweak There are a number of methods which require querying SAS for information which is returned in the LOG and then parsed out by SASPy. If there's something wrong on the SAS side and somehow the information isn't in the LOG, it can cause exceptions in the method. I've added code to catch these exceptions and just handle them like an error so processing can continue. The methods will now just fail and you can look at the log to see what the problem was.

Fixed

• Fix There were a couple methods on the SASdata object which, when returning Pandas results, didn't specify the work library explicitly on the retrieval step. So if there was a user libref assigned, it would look there instead of work for the data and not find it. All better now.

• Fix In the IOM Access Method, the Java IOM Client code does some of the transcoding between Unicode and whatever the SAS Session Encoding is. If there is a transcode failure, it throws an exception which terminates the Java process and the SAS server process. I added code to catch this trye of failure, in my Java code, and return the failure, keeping the Java and SAS processes (and SASPy) going. I don't have a way to change the behavior of the client code, so catching this and keeping things running are what I can do, which is better than it all terminating out from under SASPy.

• Fix And the big fix for this release is for df2sd() and sd2df(), fixing problems when the record length is larger than the max (32767) for the infile statement, which is used in the SAS code for these methods. This caused problems in df2sd() if the row length was larger than 32767. There was also an issue that could be hit w/ regards to this in sd2df() also, so that's addressed with this fix to. This is for all 3 access methods; STDIO, IOM and HTTP.

Removed

• None Nothing removed

[4.3.1] - 2022-07-06

Added

• Tweak Thanks to our graphics design dept for creating a cool logo for SASPy! I added the graphic to the main repo page (in the readme), and to the saspy-examples repo and to the documentation page. This has nothing to do with the code or functionality; just a cool logo for SASPy!

Changed

• Tweak Changed the prompt in the IOM access method which asks for userid/pw to use 'OMR' instead of 'IOM' since the configuration keys for user/pw are omruser/omrpw not iomuser/iompw. This has no effect on the code or processing, just changing the text of the message to better correlate with the right acronym.

• Tweak Made a change in the java IOM client code for upload/download to propagate and return the error if there was a failure during the data transfer phase. Never seen this case happen before, but if it does it should be more clear.

Fixed

• Fix Added a check in the HTTP access method in upload and download to see if the status for the HTTP call was an error and return the failure. Previously it didn't check and just returned success.

• Fix Made a change in the HTTP access method's download method to read/write chunks of the data instead of the whole file to keep from running out of, or using excess, memory in the Python process.

Removed

• None Nothing removed

[4.3.0] - 2022-05-17

Added

• New Per a user request, I added the ability to use sshpass with the STDIO over SSH access method. This allows you to authenticate with user/pw instead of having to use rsa keys (passwordless shh). There are now two new keys (see the doc) for providing the path to the sshpass executable and the parameters to use. The ssh key and other keys for this configuration stay the same.

Changed

• Enhanced The sd2df methods require multiple interactions (code submissions) to the SAS server and if any of these intermediate steps fail or have some issue, then the method invocation fails. But, I wasn't catching these intermediate problems which could cause non-obvious exceptions and tracebacks which were confusing. I've added code to catch failures for these intermediate steps and throw a more clear exception if that happens. Nothing about how the methods works has changed, just better error handling.

• Enhanced The code that checks for an 'ERROR:' in the log and issues a warning to alert you to look to see what the error was, wasn't only looking for it to start in column 1 of the log. So, it could pick up 'ERROR:' in a comment in the code or anywhere in the log. I've enhanced this to limit false positives by only flagging 'ERROR:' starting in the first column of the LOG.

• Tweak Fixed a typo in an error message.

Fixed

• Fix The COM Access Method had a bug where the encoding was being used to transcode HTML results returned to SASPy when it shouldn't, since the HTML results are already utf-8. Other code paths retrieving data from SAS did need to use the provided encoding for transcoding from SAS Session encoding to utf-8. So, this one path was changed to not try to transcode the HTML results. This was a fix for issue 454.

Removed

• None Nothing removed

[4.2.0] - 2022-03-22

Added

• None Nothing Added

Changed

• Tweak If the initial connection to SAS (a SASsession) fails then there will be a failure error provided, but there is also an exception thrown in subsequent code, trying to submit some of the initial code to gather info about the session. This was subordinate and inconsequential, yet was confusing and needed to be addressed. So, in this release, there is a specific exception being thrown instead of the one that happened to be thrown, which makes the problem more clear.

Fixed

• Fix There was an issue opened for the STDIO Access Method where if the code being submitted was longer than 128K, SASPy would deadlock with SAS due to both being single threaded and the way Pipes work; trying to flush STDIN would block in SASPy because SAS was blocked on writing to STDOUT/ERR, so they would both deadlock, not being able to read off of the other Pipes. I reworked how submit works in STDIO, so that there's a blocked amount of STDIN sent before trying to pull off of both STDOUT and STDERR, iterating till done, to circumvent this behavior. That works great, but it caused issues with ATTN handling that I support in that Access Method. I needed to rework that to compensate and it's handling it much better.

Removed

• None Nothing removed

Added

• New I added a new method, lib_path(), off the SASsession object which returns a list of the path(s) for the libref. Depending upon the engine, 'path' may mean different things. For BASE engines (Linux/PC), it's a directory or multiple directories if a concatenated libref. For database engines, it varies; may be the database name that you're connected to. On MVS, it could be then name of a bound library file, or a linux directory path.

Changed

• Tweak In the submit method of the HTTP access method, for results='text', I removed extraneous empty lines before and after the code. These resulted in extra 'return's being submitted. Removing these allows correct behavior when using saspy to interactively debug (w/ pdb) PROC PYTHON code. With the extra 'return' before and after the command you tried to submit, pdb executed extra commands. Now it's 1:1 with clean debugging using submit().

Fixed

• None No changes

Removed

• None Nothing removed

Note the Major release number being incremented! This means there is a breaking change in this release (yes, the first time so far).

Also note the addition of the CHANGELOG.md file in the repo, where all release changes will be logged, starting with this release!

The breaking change is as follows. sasdata2dataframe (sd2df or other aliases), has returned None when the SAS Data Set provided doesn't exist. Of course, trying to use that as a dataframe results in an exception. But, if you coded a check for this and conditionally didn't use it, then this change will break that code. A FileNotFoundError exception is now thrown, as that's the appropriate Pythonic thing to do in this situation.

If you happened to code this:

df = sas.sd2df('table','libref')
if df is None:
   # do something because the data set wasn't found in the SAS session

Then you would need to change that to:

try:
   df = sas.sd2df('table','libref')
except FileNotFoundError:
   # do something because the data set wasn't found in the SAS session

There is another change that I had to make for the log4j problems. If you recently used the new 'log4j' configuration key in the pervious release, you may need to update the value. SASPy V3.7.8 had log4j 2.12.2 and 2.16.0 jars, but now 2.12.4 and 2.17.1 are included instead.

So, if you happened to use log4j='2.16.0', you need to change that to log4j='2.17.1' in this release. Be aware that this can continue to change if more vulnerabilities are found and fixed in log4j.

Note also that none of the vulnerabilities are exposed by SASPy as it doesn't use log4j, so there's no actual problem with these regardless of their version.

This release addresses the current issues with log4j by providing both the fixed version (https://logging.apache.org/log4j/2.x/index.html) of log4j that still supports Java 7, saspy's minimum version of Java, using log4j 2.12.2, but also includes the 2.16.0 versions which you can choose to use, if you want and have Java 8 or higher. There is a new configuration definition key, 'log4j', that you can provide in either your config definition or on the SASsession() method (using 2.12.2 is the default - you don't need to provide this unless trying to use 2.16.0). This is documented in the SASPy V3.7.8 documentation under configuring the IOM using Java section. Note that SASPy doesn't expose any log4j vulnerabilities, regardless of the log4j version, because it doesn't use or even initialize log4j. But, having these version in the deployment will mitigate security scans and 'false positives'. There is more discussion on all of this in the #429 issue. Feel free to read more there, an respond if you want or have any other questions or concerns.

This version updates the log4j jar versions to patch the security issues reported in the last few days. The details are found in #429.

Note that this change now causes the minimum Java version to move to Java 8, from the previous minimum of Java 7 (log4j version 2.12, which is what saspy included until now, was the highest version still supporting java 7). Also note that saspy does not expose any of the the log4j vulnerabilities, so it's not necessary nor required to move to this version. None of the known, or still to be found, log4j vulnerabilities are exposed by saspy because it neither uses nor even initializes log4j, so no logging is done via log4j; neither internal nor any containing user provided content.

This release just has 2 fixes in it, from the previous version. Fixes for issues #421 and #422.

This release has a user requested enhancement and another user contributed enhancement. Both of these are doc'ed in the Advanced Topics section of the saspy doc (https://sassoftware.github.io/saspy/advanced-topics.html#).

The first is an enhancement to support the Python logging facility, instead of saspy using print() to write messages to the output. This is a great feature and provides a lot of way to configure it yo your needs; from setting what level of messages to get, and to specifying whether the show up in the output or in a file if you like. You can even provide formatting, like timestamps prepended to the messages or whatever you want.

The other is a contribution (#401) which allows the SASsession object to be used as a Context Manager, like in the 'with' statement. This automatically ends the session once the 'with' block of code has completed. Very convenient.

Another thing is for getting the Script directory (containing the user contributed script which was added in V3.7.3) actually installed with SASPy. I forget to add that directory to the selection criteria for what get's deployed when you pip install. My bad on that one. But, now the run_sas.py script will be available in the saspy deployment when you install it.

Other than that there were some doc updates and changes and a couple other tweaks, but the above are the main things added to this release.

This release has just two enhancements. The first is to support embedded quotes in SAS variable and table names. SAS supports this via the n-literal syntax (which SASPy has always used in it's code gen). For example: 'other"name'n or "other''name"n. or 'other''name'n you can't tell here (embeded quotes are a pain everywhere), but some of those are 2 single quotes a some one double quote ... (took me 30 min just to get these examples to resolve in this editor!)

The n-literal syntax supports all kinds of things, not just quotes; blanks, special chars,... So saspy has generated that syntax all along, but I misses the embedded quotes case and it tangles up parsing because they require special handling compared to anything else in the n-lit syntax (have to double single quotes, double quote single quotes, . bla bla bla). So this version should handle all of this appropriately. Here's a SAS example using this:

data "cars''2"n; set sashelp.cars( rename=( Make='Make''s'n Model='Model"''s'n Cylinders="Cylinder''s"n horsepower="HP''s"n ) ); run;

For methods that take table or variable names explicitly, you can use Python string syntax to provide then; no need for the n-lit syntax; I generate the corect SAS code for those. But some methods take arbitrary string that are embedded directly into the code w/ no massaging from SASPy (no idea what the syntax is to be able to do so), so for those cases, you have to provide correct SAS (including n-literal as necessary) syntax yourself:

For known table or variable name just use Python string syntax for embedded quotes cars = sas.sasdata("cars'2") cars.heatmap('msrp',"hp's")

For arbitrary strings that are just included into the code, use SAS syntax cars.where(''' "HP's"n > 400 ''').head() stat_results = stat.reg(model=" 'hp''s'n = 'Cylinder''s'n EngineSize ",data=cars)

The other enhancement is is specific to Viya 4 (the HTTP access method), for authenticating to Viya. Originally user/pw were the only way. Right before SGF 2021, SSO (Single Sign On) was implemented which uses a client_it, client_secret and authcode, and a client_id/secret was created for SASPy which now ships w/ Viya 4, so all you actually need to do is just provide the authcode, which is like when you log onto a secure account and it text's you a code to type in to prove it's your, even though you've already authenticated, as an extra security mechanism. So that's already there. But what's new in this release is another variation on that.

The HTTP access to Viya 4 now support providing a JWT witch is an authentication token from other than SASLogon, which (if both systems are configured to share), can be used to acquire the SASLongon authentication token that is required for all API calls into Viya. This allows, for instance, an Azure ML instance to connect to Viya by providing the Azure auth token to saspy which then uses that (as opposed to user/pw or authcode) to get the SASLogon auth token. This config option is jwt= on the SASsesion() method. Support for passing a SASLogon token directly in is also available (meaning you've already somehow made calls to SASLogon yourself, or some other tool, and have a valid authtoken), via authtoken= on the SASsession() method. Both of these are really expected to be used under the covers by other tools/processes, but can be used by advanced users who can interface to the authenticating providers and acquire these tokens themselves directly.

This release has a few enhancements. The IOM access method uses the Java IOM client (some jars files), and the log4j.jar in that is pretty old; 1.2. That version if past EOL, and raises flags on some security scans. So, I've upgraded to 2.12, which isn't the absolute newest, but it's close and the highest version that still supports Java7; one higher and Java8 is the minimum. Since saspy still supports back to Java7, and it isn't EOL till end of this year, I've gone w/ log4j 2.12. Maybe upgrade it in the future after java7 is dead.

Another cool thing is a user contributed script (see it in the new saspy/scripts dir, and pr #392 ) for batch submitting a .sas file, similar to how you could batch submit a .sas file with SAS from the command line. Check it out.

Another little enhancements is the the char_lengths parm of df2sd, which now accepts only the subset of columns you want to overrise; the others get calculated per usual. Before you had to provide the whole list.

There's a fix for SAS other missing values ('.A'-'.Z' and '._') from issue #390, so now those are all NaN,s when transferred, as they should have been.

Some doc changes for things, and that's about it.

This is a minor release with just a couple of enhancements. One is a user contribution to enhance the saslib() method to support assigning concatenated libraries. That's a nice enhancement; just provide a list of paths instead of a single one and viola; you've got a concatenated libref. There's also a new configurations key on the HTTP access method; 'inactive'. The compute service in Viya has a 15 minute inactivity time out, by default, for a Session. This option allows you to set the number of minutes for this inactivity timeout. The default for this new option, however, is configured to be 120 minutes (2 hours), so you get that benefit without even needing to set this yourself. If you've had an HTTP Session and stepped away for 15 minutes then got a 404 on the next thing you submit, this will be a welcome improvement.

This release has enhancements for the HTTP Access Method which connects to SAS in an Viya deployment. Prior to this release, only user/pw authentication was available. But there is a different mechanism for connecting in a SSO (Single Sign On) configurations of Viya. This requires 'clients' to be defined which provide scope and authorization groupings, ... An admin creates these as part of post-deployment so that an end user can then use their credentials to acquire one time auth codes used to signon (instead of their user/pw), based upon the client_id and client_secret previously set up. There are new config definition keys for these attributes and prompting in place when necessary for the authcode if it's not provided on the SASsession() method.

This release has some major performance enhancements to the sasdata2dataframe (sd2df) methods in all 3 access methods; STDIO (including over SSH), IOM, and HTTTP. This is for all 3 methods as well; MEM, CSV, DISK. In fact, MEM and DISK are now one in the same, with 40% improvements in DISK for remote cases which means virtually infinite improvement for MEM for large datasets. MEM was too slow for large data, which is why CSV and DISK were implemented. Now, CSV and DISK aren't needed, but of course, are still there, and just work faster. MEM is still the default, and is now as fast. The key to this is that I no longer actually write the data being streamed over from SAS to a disk file, to then call pandas read_csv() to read the file and create the dataframe. Instead, I directly stream the data into read_csv() with no disk overhead and a direct feed into read_csv(). There is only one case where disk is still used, as that's just a bit faster. That is IOM Local with CSV. In that case, SAS has to write the csv file to disk before being able to stream it, so in that case, skipping that and just pointing read_csv() at the file is a little faster. All other cases I've tested are faster with this new scheme. So, this release is the performance release for moving SAS data to Pandas.

This version has a user requested enhancement to issue a warning where there is an ERROR in the SASLOG. This is done using the 'warnings' module, which is part of (and documented in) the Python standard Library. You can configure warnings to have these be issued never/sometimes/always, or even throw exceptions. It's actually kinda cool. I've also added some examples in the Advanced Topics section of the doc. A warning will also now be issued when df2sd() identifies that there is an index (row label) not the default of RangeIndex. This is when you've set_index() to a column. In this case, that indexed column isn't a dataframe.column anymore, so it wouldn't be transferred over to the SAS data set. This warning lets you know that column won't be transferred. You can simply use df2sd(df.reset_index(), ...) to allow that column to be transferred. There is also a fix for the STDIO over SSH from a Windows client where reading the log and lst could get 'out of sequence' and result in missing output and partial xml being displayed. So that is fixed in there too. There's also one more enhancement request in here for the disconnect() method in the IOM Access Method's Remote case. Upon disconnecting, the token needed to reconnect is now provided via an attribute off of the SASsession object; sas.reconuri. This can be now be provided via on the SASsession() method to be able to reconnect from a different Python process, via: sas = saspy.SASsession(reconuri=''), where you need to use the value from the previous session. this is not needed in the usual case where you're reconnecting from the same session; that still works as it always has. This is only for a situation where you have a new Python process you're using and want to reconnect.

This release just has a couple fixes for MVS; connecting to a Workspace server running on MVS (Mainframe), via Remote IOM. I had caused a regression at some time w/ sd2df() and sd2df_DSIK(), as well as a type issue w/ encodings. This fixes all that and enhances the encoding validation code to support the default 1047 encoding to use cp500 which is all that is really needed for what little transcoding is required. So, the default just works w/out you needing to specify the encoding.

This release just has some minor fixes and changes. There are a number or places in the HTTP access method I've needed to add checks to catch unexpected failures and disconnects and such, and provide a better exception; all failure cases, just better messages. A couple fixes for issues that were found since last release. I stripped FF (Form Feed) chars out of the log, as they are just annoying in terminal mode, and they keep the log from being displayed in some UI/IDEs. Unless you're sending the output to an actual line printer, you don't need them. I also changed the default value for method= in submitLST() from 'listonly' to 'listorlog', as, after hundreds of times using the method, I only ever want it to be 'listorlog'. The symmetry of 'listonly', with submitLOG(); getting the log or nothing, and the LST or nothing, was highly overrated. I don't use submitLST() if I don't expect to get output. And if I don't get output, I then go look at the LOG to see why! So, I switched the default to be the one that's realistically useful. You could say that's a breaking change, but I'd argue it's a fixing change :) I should have made that the default to start with. That's about it for this version.

This release has three new features. Two new methods to compliment dirlist() and file_info() for server side file system access: file_copy() and file_delete(). Those are pretty self explanatory, and are in the API doc. The other new feature is from Issue #353, adding the ability to use the STDIO over SSH (only over SSH) from a Windows client. STDIO (SSH or not) can only access a stand alone Linux SAS install. But this feature allows the Python process to be on a windows client as opposed to a Linux client; which has been the requirement since I first wrote it.

This release just has a few fixes/enhancements. One fix is to the HTTP access method, disabling the encoding option. Unlike the other access methods, where saspy has to transcode everything between pythons utf-8 and the SAS session encoding, the HTTP API being used requires everything to be utf-8 and it transcodes to/from SAS session encoding. Easy fix for saspy, and it's in here in this release. There's a new user contributed method, validvarname, which renames the columns of your dataframe to match the SAS 'validvarname' constraints for however that option is set. This would be used prior to passing that dataframe to df2sd(). So now if you have that set to a more restrictive value, you can use this method to convert your column names to be compliant. This is something Access Engines do, based up the option setting, when accessing database table, which allow names that don't comply. There are a couple other fixes, one in COM to fix a case where only missing values in a numeric SAS data set becomes a char type in the dataframe instead of numeric when importing in sd2df(). And another to allow overrides for index_col= and engine=, also in sd2df. These had been hard coded in sd2df, but can not be user specified.

This release just has 3 fixes in it. One to fix a regression for STDIO when a dataframe is empty; still create the empty dataset. That was a break in 3.6.0. Fix an import error that can happen; depends on maybe what other modules you have installed? Either way, it's fixed for all cases now. That was in IOM access method. And, there was a code path in tail() where an error could happen. Fixed that too in this release. That's all.

This release has a number of enhancements for df2sd (dataframe2sasdata). This started as performance changes for #326 and included more for #332. The main change had to do with calculating the lengths for char columns of the dataframe, which has to be done to declare the correct byte lengths for the corresponding SAS variables in the SAS data set being created. With a DF having 150 million rows and 100 char columns, this step was taking way too long. I separated out this step from df2sd (df_char_lengths()) so it can be called independently (by the user or by the access methods df2sd), returning a dict with the char column names and lengths. I also made enhancements to this routine to be able to shortcut some of the time calculating lengths so it could be quicker. df2sd can take these options for when it calls this internally, but it can also take a dict with the char column names and lengths (that is returned by that method, or you can just code that yourself so that the metadata calculation step can be done once, or skipped altogether and just go to the data transfer. I also enhanced the data transfer step in the STDIO access method significantly too. Handling transcoding failure is now handled in the data transfer step (though it can still be caught in the length calc routine if wanted), and you now have the option of replacing chars that can't be transcoded, with the replacement char, instead of failing. So there's a lot of new functionality and performance improvements that can be tapped into in this version for df2sd. The default behaviors, for the most part, are still the same as they were. So if df2sd seems too slow, there are a number of ways to improve it's performance in this version, by tweaking these options. Oh, and I almost forgot, df2sd also now has an outdsopts={...} parameter which allows you to specify key=value output data set options for the data set being created: for instance, compress=, encoding=, index=, outrep=, replace=, rename= ...

This release only has a few enhancements in it. The symexist and symget methods have a fix for a macro named 'id' as there was a parsing issue with that one specific name. symput has a new option so you can specify the specific SAS quoting function to use, as the default doesn't always work. SASPy tries to remove as many SAS'isms from the python code, by defaulting behavior for the more usual cases. But, SAS has no end of options and varying statements to tweak and special case things, that, sometimes you just have to be able to specify specific SAS syntax for certain cases. The other fix in this release was due to the SPDE engine not supporting (case in point from above!) the common OBS and FIRSTOBS data set options. It has it's own names for these, and sometimes, will allow OBS= and sometimes not. So, I've added support for this in the methods where I generate any of these options (head/tail) so that they work right with this engine as well as others. Also, as part of this, I found that head and tail weren't exactly honoring these options when specified in the DSOPTS of the SASdata object. Now the head and tail set is accurately based off of the result set defined by these options when in DSOPTS.